Nova Spivack - Minding the Planet » Memes & Memetics

What's After the Real Time Web?

Nova — Wed, 28 Oct 2009 04:08:01 +0000

In typical Web-industry style we’re all focused minutely on the leading trend-of-the-year, the real-time Web. But in this obsession we have become a bit myopic. The real-time Web, or what some of us call “The Stream,” is not an end in itself, it’s a means to an end. So what will it enable, where is it headed, and what’s it going to look like when we look back at this trend in 10 or 20 years?

In the next 10 years, The Stream is going to go through two big phases, focused on two problems, as it evolves:

Web Attention Deficit Disorder. The first problem with the real-time Web that is becoming increasingly evident is that it has a bad case of ADD. There is so much information streaming in from so many places at once that it’s simply impossible to focus on anything for very long, and a lot of important things are missed in the chaos. The first generation of tools for the Stream are going to need to address this problem.
Web Intention Deficit Disorder. The second problem with the real-time Web will emerge after we have made some real headway in solving Web attention deficit disorder. This second problem is about how to get large numbers of people to focus their intention not just their attention. It’s not just difficult to get people to notice something, it’s even more difficult to get them to do something. Attending to something is simply noticing it. Intending to do something is actually taking action, expending some energy or effort to do something. Intending is a lot more expensive, cognitively speaking, than merely attending. The power of collective intention is literally what changes the world, but we don’t have the tools to direct it yet.

The Stream is not the only big trend taking place right now. In fact, it’s just a strand that is being braided together with several other trends, as part of a larger pattern. Here are some of the other strands I’m tracking:

Messaging. The real-time Web aka The Stream is really about messaging in essence. It’s a subset of the global trend towards building a better messaging layer for the Web. Multiple forms of messaging are emerging, from the publish-and-subscribe nature of Twitter and RSS, to things like Google Wave, Pubsubhubub, and broadcast style messaging or multicasting via screencast, conferencing and media streaming and events in virtual worlds. The effect of these tools is that the speed and interactivity of the Web are increasing — the Web is getting faster. Information spreads more virally, more rapidly — in other words, “memes” (which we can think of as collective thoughts) are getting more sophisticated and gaining more mobility.
Semantics. The Web becomes more like a database. The resolution of search, ad targeting, and publishing increases. In other words, it’s a higher-resolution Web. Search will be able to target not just keywords but specific meaning. For example, you will be able to search precisely for products or content that meet certain constraints. Multiple approaches from natural language search to the metadata of the Semantic Web will contribute to increased semantic understanding and representation of the Web.
Attenuation. As information moves faster, and our networks get broader, information overload gets worse in multiple dimensions. This creates a need for tools to help people filter the firehose. Filtering in its essence is a process of attenuation — a way to focus attention more efficiently on signal versus noise. Broadly speaking there are many forms of filtering from automated filtering, to social filtering, to personalization, but they all come down to helping someone focus their finite attention more efficiently on the things they care about most.
The WebOS. As cloud computing resources, mashups, open linked data, and open API’s proliferate, a new level of aggregator is emerging. These aggregators may focus on one of these areas or may cut across them. Ultimately they are the beginning of true cross-service WebOS’s. I predict this is going to be a big trend in the future — for example instead of writing Web apps directly to various data and API’s in dozens of places, just write to a single WebOS aggregator that acts as middleware between your app and all these choices. It’s much less complicated for developers. The winning WebOS is probably not going to come from Google, Microsoft or Amazon — rather it will probably come from someone neutral, with the best interests of developers as the primary goal.
Decentralization. As the semantics of the Web get richer, and the WebOS really emerges it will finally be possible for applications to leverage federated, Web-scale computing. This is when intelligent agents will actually emerge and be practical. By this time the Web will be far too vast and complex and rapidly changing for any centralized system to index and search it. Only massively federated swarms of intelligent agents, or extremely dynamic distributed computing tools, that can spread around the Web as they work, will be able to keep up with the Web.
Socialization. Our interactions and activities on the Web are increasingly socially networked, whether individual, group or involving large networks or crowds. Content is both shared and discovered socially through our circles of friends and contacts. In addition, new technologies like Google Social Search enable search results to be filtered by social distance or social relevancy. In other words, things that people you follow like get higher visibility in your search results. Socialization is a trend towards making previously non-social activities more social, and towards making already-social activities more efficient and broader. Ultimately this process leads to wider collaboration and higher levels of collective intelligence.
Augmentation. Increasingly we will see a trend towards augmenting things with other things. For example, augmenting a Web page or data set with links or notes from another Web page or data set. Or augmenting reality by superimposing video and data onto a live video image on a mobile phone. Or augmenting our bodies with direct connections to computers and the Web.

If these are all strands in a larger pattern, then what is the megatrend they are all contributing to? I think ultimately it’s collective intelligence — not just of humans, but also our computing systems, working in concert.

Collective Intelligence

I think that these trends are all combining, and going real-time. Effectively what we’re seeing is the evolution of a global collective mind, a theme I keep coming back to again and again. This collective mind is not just comprised of humans, but also of software and computers and information, all interlinked into one unimaginably complex system: A system that senses the universe and itself, that thinks, feels, and does things, on a planetary scale. And as humanity spreads out around the solar system and eventually the galaxy, this system will spread as well, and at times splinter and reproduce.

But that’s in the very distant future still. In the nearer term — the next 100 years or so — we’re going to go through some enormous changes. As the world becomes increasingly networked and social the way collective thinking and decision making take place is going to be radically restructured.

Social Evolution

Existing and established social, political and economic structures are going to either evolve or be overturned and replaced. Everything from the way news and entertainment are created and consumed, to how companies, cities and governments are managed will change radically. Top-down beaurocratic control systems are simply not going to be able to keep up or function effectively in this new world of distributed, omnidirectional collective intelligence.

Physical Evolution

As humanity and our Web of information and computatoins begins to function as a single organism, we will evolve literally, into a new species: Whatever is after the homo sapien. The environment we will live in will be a constantly changing sea of collective thought in which nothing and nobody will be isolated. We will be more interdependent than ever before. Interdependence leads to symbiosis, and eventually to the loss of generality and increasing specialization. As each of us is able to draw on the collective mind, the global brain, there may be less pressure on us to do things on our own that used to be solitary. What changes to our bodies, minds and organizations may result from these selective evolutionary pressures? I think we’ll see several, over multi-thousand year timescales, or perhaps faster if we start to genetically engineer ourselves:

Individual brains will get less good at things like memorization and recall, calculation, reasoning, and long-term planning and action.
Individual brains will get better at multi-tasking, information filtering, trend detection, and social communication. The parts of the nervous system involved in processing live information will increase disproportionately to other parts.
Our bodies may actually improve in certain areas. We will become more, not less, mobile, as computation and the Web become increasingly embedded into our surroundings, and into augmented views of our environments. This may cause our bodies to get into better health and shape since we will be less sedentary, less at our desks, less in front of TV’s. We’ll be moving around in the world, connected to everything and everyone no matter where we are. Physical strength will probably decrease overall as we will need to do less manual labor of any kind.

These are just some of the changes that are likely to occur as a result of the things we’re working on today. The Web and the emerging Real-Time Web are just a prelude of things to come.

My Burma Meme Spreads to 17,000 Web Pages in just one week!

Nova — Thu, 04 Oct 2007 07:01:38 +0000

I’ve been tracking the progress of my Burma protest meme. In just under one week it has spread to almost 17,000 web pages and it continues to grow. (For the latest number, click here). It’s great to see the blogosphere pick this up, and I’m glad to be able to do something to help raise awareness of this important human rights issue.

This meme is also an example of an interesting new way to spread content on the Web — whether for a protest or an ad or any other kind of announcement. It’s kind of like a chain letter, but via weblogs. There are many different ways to structure these memes with varying levels of virality and benefit to participants. For some earlier work I’ve done on meme propagation on the Web see my GoMeme experiments from a few years ago. In those experiments I created a series of memes that spread widely through the blogosphere, based on different viral messages, surveys, and benefits to participants. Other people then tracked the statistics of the memes as they spread. It turned out to be a very interesting study of superdistribution of content along social networks.

Networked Genome — New Finding Shatters Current Thinking

Nova — Wed, 15 Aug 2007 18:08:56 +0000

A new finding has discovered that the human genome may be highly networked. That is, genes do not operate in isolation, but rather they are networked together in a far more complex ecosystem than previously thought. It may be impossible to separate one gene from another in fact. This throws into question not only our understanding of genetics and the human genome, but also the whole genomics industry, which relies heavily on the idea that genes and drugs based on them can be patented:

The principle that gave rise to the biotech industry promised
benefits that were equally compelling. Known as the Central Dogma of
molecular biology, it stated that each gene in living organisms, from
humans to bacteria, carries the information needed to construct one
protein.

The scientists who invented recombinant DNA in 1973 built their
innovation on this mechanistic, "one gene, one protein" principle.

Because donor genes could be associated with specific functions,
with discrete properties and clear boundaries, scientists then believed
that a gene from any organism could fit neatly and predictably into a
larger design – one that products and companies could be built around,
and that could be protected by intellectual-property laws.

This presumption, now disputed, is what one molecular biologist calls "the industrial gene."

"The industrial gene is one that can be defined, owned, tracked,
proven acceptably safe, proven to have uniform effect, sold and
recalled," said Jack Heinemann, a professor of molecular biology in the
School of Biological Sciences at the University of Canterbury in New
Zealand and director of its Center for Integrated Research in Biosafety.

In the United States, the Patent and Trademark Office allows genes
to be patented on the basis of this uniform effect or function. In
fact, it defines a gene in these terms, as an ordered sequence of DNA
"that encodes a specific functional product."

In 2005, a study showed that more than 4,000 human genes had already
been patented in the United States alone. And this is but a small
fraction of the total number of patented plant, animal and microbial
genes.

In the context of the consortium’s findings, this definition now
raises some fundamental questions about the defensibility of those
patents.

If genes are only one component of how a genome functions, for
example, will infringement claims be subject to dispute when another
crucial component of the network is claimed by someone else?

Might owners of gene patents also find themselves liable for
unintended collateral damage caused by the network effects of the genes
they own?

And, just as important, will these not-yet-understood components of
gene function tarnish the appeal of the market for biotech investors,
who prefer their intellectual property claims to be unambiguous and
indisputable?

While no one has yet challenged the legal basis for gene patents,
the biotech industry itself has long since acknowledged the science
behind the question.

"The genome is enormously complex, and the only thing we can say
about it with certainty is how much more we have left to learn," wrote
Barbara Caulfield, executive vice president and general counsel at the
biotech pioneer Affymetrix, in a 2002 article on Law.com called "Why We
Hate Gene Patents."

"We’re learning that many diseases are caused not by the action of
single genes, but by the interplay among multiple genes," Caulfield
said. She noted that just before she wrote her article, "scientists
announced that they had decoded the genetic structures of one of the
most virulent forms of malaria and that it may involve interactions
among as many as 500 genes."

Even more important than patent laws are safety issues raised by the
consortium’s findings. Evidence of a networked genome shatters the
scientific basis for virtually every official risk assessment of
today’s commercial biotech products, from genetically engineered crops
to pharmaceuticals.

Read the rest here

Intelligence is in the Connections

Nova — Tue, 20 Feb 2007 16:26:58 +0000

Google’s Larry Page recently gave a talk to the AAAS about how Google is looking towards a future in which they hope to implement AI on a massive scale. Larry’s idea is that intelligence is a function of massive computation, not of “fancy whiteboard algorithms.” In other words, in his conception the brain doesn’t do anything very sophisticated, it just does a lot of massively parallel number crunching. Each processor and its program is relatively “dumb” but from the combined power of all of them working together “intelligent” behaviors emerge.

Larry’s view is, in my opinion, an oversimplification that will not lead to actual AI. It’s certainly correct that some activities that we call “intelligent” can be reduced to massively parallel simple array operations. Neural networks have shown that this is possible — they excel at low level tasks like pattern learning and pattern recognition for example. But neural networks have not proved capable of higher level cognitive tasks like mathematical logic, planning, or reasoning. Neural nets are theoretically computationally equivalent to Turing Machines, but nobody (to my knowledge) has ever succeeded in building a neural net that can in practice even do what a typical PC can do today — which is still a long way short of true AI!

Somehow our brains are capable of basic computation, pattern detection and learning, simple reasoning, and advanced cognitive processes like innovation and creativity, and more. I don’t think that this richness is reducible to massively parallel supercomputing, or even a vast neural net architecture. The software — the higher level cognitive algorithms and heuristics that the brain “runs” — also matter. Some of these may be hard-coded into the brain itself, while others may evolve by trial-and-error, or be programmed or taught to it socially through the process of education (which takes many years at the least).

Larry’s view is attractive but decades of neuroscience and cognitive science have shown conclusively that the brain is not nearly as simple as we would like it to be. In fact the human brain is far more sophisticated than any computer we know of today, even though we can think of it in simple terms. It’s a highly sophisticated system comprised of simple parts — and actually, the jury is still out on exactly how simple the parts really are — much of the computation in the brain may be sub-neuronal, meaning that the brain may actually a much much more complex system than we think.

Perhaps the Web as a whole is the closest analogue we have today for the brain — with millions of nodes and connections. But today the Web is still quite a bit smaller and simpler than a human brain. The brain is also highly decentralized and it is doubtful than any centralized service could truly match its capabilities. We’re not talking about a few hundred thousand linux boxes — we’re talking about hundreds of billions of parallel distributed computing elements to model all the neurons in a brain, and this number gets into the trillions if we want to model all the connections. The Web is not this big, and neither is Google.

One reader who commented on Larry’s talk made an excellent point on what this missing piece may be: “Intelligence is in the connections, not the bits.”The point is that most of the computation in the brain actually takesplace via the connections between neurons, regions, and perhapsprocesses. This writer also made some good points about quantumcomputation and how the brain may make use of it, a view that forexample Roger Penrose and others have spent a good deal of time on.There is some evidence that brain may make use of microtubules andquantum-level computing. Quantum computing is inherently about fields,correlations and nonlocality. In other words the connections in thebrain may exist on a quantum level, not just a neurological level.

Whether quantum computation is the key or not still remains to bedetermined. But regardless, essentially, Larry’s approach is equivalentto just aiming a massively parallel supercomputer at the Web and hopingthat will do the trick. Larry mentions for example that if allknowledge exists on the Web you should be able to enter a query and geta perfect answer. In his view, intelligence is basically just search ona grand scale. All answers exist on the Web, and the task is just tomatch questions to the right answers. But wait? Is that all thatintelligence does? Is Larry’s view too much of an oversimplification?Intelligence is not just about learning and recall, it’s also aboutreasoning and creativity. Reasoning is not just search. It’s unclearhow Larry’s approach would address that.

In my own opinion, for global-scale AI to really emerge the Web has toBE the computer. The computation has to happen IN the Web, betweensites and along connections — rather than from outside the system. Ithink that is how intelligence will ultimately emerge on a Web-widescale. Instead of some Google Godhead implementing AI from afar for thewhole Web, I think it is more likely that every site, app and person onthe Web will help to implement it. It will be much more of a hybridsystem that combines decentralized human and machine intelligences andtheir interactions along data connections and social relationships. Ithink this may emerge from a future evolution of the Web that providesfor much richer semantics on every piece of data and hyperlink on theWeb, and for decentralized learning, search, and reasoning to takeplace within every node on the Web. I think the Semantic Web is anecessary technology for this to happen, but it’s only the first step.More will need to happen on top of it for this vision to reallymaterialize.

My view is more of an “agent metaphor” for intelligence — perhaps itis similar to Marvin Minsky’s Society of Mind ideas. I think that mindsare more like communities than we presently think. Even in our ownindividual minds for example we experience competing thoughts, multiplethreads, and a kind of internal ecology and natural selection of ideas.These are not low-level processes — they are more like agents — theyare actually each somewhat “intelligent” on their own, they seem to besomewhat autonomous, and they interact in intelligent almost socialways.

Ideas seem to be actors, not just passive data points — they arecompeting for resources and survival in a complex ecology that existsboth within our individual minds and between them in socialrelationships and communities. As the theory of memetics proposes,ideas can even transport themselves through language, culture, andsocial interactions in order to reproduce and evolve from mind to mind.It is an illusion to think that there is some central self or “I” thatcontrols the process (that is just another agent in the community infact, perhaps one with a kind of reporting and selection role).

I’m not sure the complex social dynamics of these communities ofintelligence can really be modeled by a search engine metaphor. Thereis a lot more going on than just search. As well as communication andreasoning between different processes, there may in fact be feedbackacross levels from the top-down as well as the from the bottom-up.Larry is essentially proposing that intelligence is a purely bottom-upemergent process that can be reduced to search in the ideal, simplestcase. I disagree. I think there is so much feedback in every directionthat medium and the content really cannot be separated. The thoughtsthat take place in the brain ultimately feedback down to the neuralwetware itself, changing the states of neurons and connections –computation flows back down from the top, it doesn’t only flow up fromthe bottom. Any computing system that doesn’t include this kind offeedback in its basic architecture will not be able to implement trueAI.

In short, Google is not the right architecture to truly build a globalbrain on. But it could be a useful tool for search andquestions-and-answers in the future, if they can somehow keep up withthe growth and complexity of the Web.

'Bemes' are Defining the Blogosphere

Nova — Tue, 20 Feb 2007 03:29:57 +0000

Tom Hayes has an interesting post in which he coins the word ‘beme" to mean a meme that spreads in the blogosphere.

Michael Malone’s ABC News column on Thursday mentioning "bemes" has certainly produced a lot of interest. Originally, I coined the word beme
to describe a meme propagated by blogs and bloggers. Now I can see
that the turn of phrase has a much bigger potential to capture the
rapidly-moving cultural touchstones of the Bubble Generation.

As you may know, "meme" was first defined by Richard Dawkins in 1976
as "a unit of cultural information" spread from one mind to another.
In other words, a viral idea that eventually becomes common knowledge.

Fast forward three decades, and it seems to me that technology has turbo-charged the meme process. Looking for the juste mot
to describe a "purposeful" meme fed into the vast human network of the
Internet, either by blog, email, video, phonecast, social media or
other viral means, beme seems to fit the bill.

A beme is a turbo-charged meme made possible entirely by the
existence of the network affect. A beme can be impactful because it is
lurid–a photo of a panty-less Britney Spears, or humorous–a
whimisical video of the band OKGO on treadmills, or gut-wrenching–the
sad tirade by comedian Michael Richards. A beme can cement an idea
with the public in a way that cannot be legislated or regulated. No
legal effort by Cisco to enforce a trademark, for example, will make
the public unlearn that Apple produces the iPhone.

A meme is old media, a beme is new media.

A meme takes off by accident, a beme by design.

A meme can take years to surface, a beme hours.

Interesting Idea: Start a Magazine that is a Wiki

Nova — Mon, 04 Sep 2006 22:13:56 +0000

I was reading this article in Wired magazine about wikis, where the article itself is a wiki that the readers can contribute to — and an idea occurred to me. What if you could make an entire magazine that was in a fact a wiki? This magazine would be published online via a Website running a wiki engine. Every issue would be by and for the community of readers. There would be an editorial group among the readers that would decide what to write articles about for the next issue of the magazine, and then the community would work to write the articles. To get into the editorial group, remain there, and have a vote as an editor, a community member would have to make a certain number of (non-spurrious) contributions to articles on an ongoing basis (and/or maintain a certain reputation in the community as measured in some other manner).

I can imagine this idea taking off and a lot of these "wikazines" forming around various subject areas. It makes sense that communities of people who are interested in subjects could help to research and write about them. Of course in such communities there would be some people who put more effort in than others, and some who were more like readers or lurkers. But it would still be much more involving than old "one-way media."

In some ways communities like Digg simulate this — people essentially vote on what is interesting and this filters up to become the featured content on the site. But that is still one step removed from the creative process itself — only the readers participate, not the content authors. What’s interesting about this proposal is that it blurs the distinction between an author and a reader, and provides a way for a magazine to be truly emergent and community-driven. OK, I’m too busy to start this, but I hope someone out there on the lazyweb takes this idea and runs with it. Please let me know if you find examples of this.

Radar Networks is Seeking Search Engineers for Large-Scale Web Mining Initiative

Nova — Tue, 29 Aug 2006 19:12:43 +0000

My company, Radar Networks, is building a very large dataset by crawling and mining the Web. We then apply a range of new algorithms to the data (part of our secret sauce) to generate some very interesting and useful new information about the Web. We are looking for a few experienced search engineers to join our team — specifically people with hands-on experience designing and building large-scale, high-performance Web crawling and text-mining systems. If you are interested, or you know anyone who is interested or might be qualified for this, please send them our way. This is your chance to help architect and build a really large and potentially important new system. You can read more specifics abour our open jobs here.

I'm Going to Start Blogging About Radar Networks Here

Nova — Sun, 27 Aug 2006 04:16:12 +0000

I
haven’t blogged very much about my stealth startup, Radar Networks,
yet. At the most, I’ve made a few cryptic posts and announcements in the past, but we’ve been keeping things pretty quiet. That’s been a conscious decision because we have been working
intensively on R&D and we just weren’t ready to say much yet.

Unlike
some companies which have done massive and deliberate hype about unreleased vapor software, we
really felt it would be better to just focus on our work and let it
speak for itself when we release it.

The fact is we have been working quietly for several years on something really big, and really hard. It hasn’t always been easy — there have been some technical challenges that took a long time to overcome. And it took us a long time to find VC’s daring enough to back us.

The thing is, what we are making is not a typical Web 2.0 "build it and flip it in 6 months" kind of project. It’s deep technology that has long-term infrastructure-level implications for the Web and the future of content. And until recently we really didn’t even have a good way to describe it to non-techies. So we just focused on our work and figured we would talk about it someday in the future.

But perhaps I’ve erred on the side of
caution — being so averse to gratuitous hype that I have literally said almost
nothing publicly about the company. We didn’t even issue a press release about our Series A round (which
happened last April — I’ll be adding one to our new corporate site, which launches on Sunday night however, for historical purposes), and until today, our site at Radar has been just
a one-page placeholder with no info at all about what we are doing.

But something happened that changed my mind about this recently. I had lunch with my friend Munjal Shah, the CEO of Riya. Listening to Munjal tell his stories about how he has blogged so openly about Riya’s growth, even from way before their launch, and how that has provided him and his team with amazingly valuable community feedback, support, critiques, and new ideas, really got me thinking. Maybe it’s time Radar Networks started telling a little more of its story? It seems like the team at Riya really benefitted from being so open. So although, we’re still in stealth-mode and there are limits to what we can say at this point, I do think there are some aspects we can start to talk about, even before we’ve launched. And besides that our story itself is interesting — it’s the story of what it’s like to build and work in a deep-technology play in today’s venture economy.

So that’s what I’m going to start doing here — I’m going to start telling our story on this blog, Minding the Planet. I already have around 500 regular readers, and most of them are scientists and hard-core techies and entrepreneurs. I’ve been writing mainly about emerging technologies that are interesting enough to inspire me to post about them, and once in a while about ideas I have been thinking about. These are also subjects that are of interest to the people who read this blog. But now I’m also going to start blogging more about Radar Networks and what we are doing and how it’s going. I’ll post about our progress, the questions we have, the achievements on our team, and of course news about our launch plans. And I hope to hear from people out there who are interested in joining us when we do our private invite-only beta tests.

We’re
still quite a ways from a public launch, but we do have something
working in the lab and it’s very exciting. Our VC’s want us to launch
it now, but it’s still an early alpha and we think it needs a lot more
work (and testing) before our baby is ready to step out into the big
world out there. But it looks promising. I do think, all modesty aside
for a moment, that it has the potential to really advance the Web on a
broad scale. And it’s exciting to work on.

This post is already long enough, so I’ll finish here for the moment. In my upcoming posts I will start to talk a little bit more about the new category that Radar Networks is going to define, and some of the technologies we’re using, and challenges we’ve overcome along the way. And I’ll share some insights, and stories, and successes we’ve had.

But I’m getting ahead of myself, and
besides that, my dinner’s ready. More later.

Harnessing The Collective Mind

Nova — Mon, 27 Mar 2006 02:09:33 +0000

Today I read an interesting article in the New York Times about a company called Rite-Solutions which is using a home-grown stock market for ideas to catalyze bottom-up innovation across all levels of personnel in their organization. This is a way to very effectively harness and focus the collective creativity and energy in an organization around the best ideas that the organization generates.

Using virtual stock market systems to measure community sentiment is not a new concept but it is a new frontier. I don’t think we’ve even scratched the surface of what this paradigm can accomplish. For lots of detailed links to resources on this topic see the wikipedia entry on prediction markets. This prediction markets portal also has collected interesting links on the topic. Here is an informative blog post about recent prediction market attempts. Here is a scathing critique of some prediction markets.

There are many interesting examples of prediction markets on the Web:

Google uses a similar kind of system — their own version of a prediction market — to enable staff members to collaboratively predict the likelihood that various internal projects and events will occur on-schedule.
Yahoo also has a prediction market called BuzzGame that enables visitors to help predict technology trends.
Newsfutures Exchange is a prediction market about the news, which is powered by a commercial prediction market engine sold by a company called Newsfutures.
BlogShares, a fantasy stock market for Weblogs in which players invest virtual money in the blogs they think will gain the most audience share.
Intrade is another exchange for trading on idea futures.
The Iowa Political Futures Exchange is a prediction market that focuses on political change.
Tradesports is a prediction market around sports topics.
The Hollywood Stock Exchange is a prediction market around movies.
The Foresight Exchange is another prediction market for predicting future events.

Here are some interesting, more detailed discussions of prediction market ideas and potential features.

Another area that is related, but highly underleveraged today, are ways to enable communities to help establish whether various ideas are correct using argumentation. By enabling masses of people to provide reasons to agree or disagree with ideas, and with those reasons as well, we can automatically rate what ideas are most agreed with or disagreed with. One very interesting example of this is TruthMapping.com. Some further concepts related to this approach are discussed in this thread.

Big Thinkers' Most Dangerous Ideas

Nova — Wed, 04 Jan 2006 17:36:00 +0000

The Edge has published mini-essays by 119 "big thinkers" on their "most dangerous ideas" — fun reading.

The history of science is replete with discoveries
that were considered socially, morally, or emotionally
dangerous in their time; the Copernican and
Darwinian revolutions are the most obvious.
What is your dangerous idea? An idea you think
about (not necessarily one you originated)
that is dangerous not because it is assumed to be false, but because it might be true?

A Cool Thingy…

Nova — Mon, 10 Oct 2005 02:05:00 +0000

This is cool Click to see why. I think this idea has great value for viral, meme-based Web advertising. Just imagine: Advertisers could release really cool animations to add to sites, and site owners could add them into their sites for entertainment or humor. The animations could run ads within them as well. It’s fun. Everyone wins, everyone’s happy. And of course users can aim these animations at any other site so visitors who like it can spread it to their own sites. Very smart!!! Very Web 2.0.

Folktologies — Beyond the Folksonomy vs. Ontology Distinction

Nova — Wed, 26 Jan 2005 18:28:17 +0000

First of all I know Clay Shirky, and he’s a good fellow. But he’s simply wrong about his claim that "tagging" (of the flavor that is appearing on del.icio.us — what I call "social tagging") is inherently better than the use of formal ontologies. Clay favors the tagging approach because it is bottom-up and emergent in nature, and he argues against ontologies because pre-specification cannot anticipate the future. But this is a simplistic view of both approaches. One could just as easily argue against tagging systems because they don’t anticipate the future — they are shortsighted, now-oriented systems that fail to capture the "big picture" or to optimally organize resources for the long-term. Their saving grace is that over time they do (hopefully) self-organize and prune out the chaff, but that depends both on the level of participation and the quality of that participation.

Tagging is certainly useful — and indeed collaborative authoring,
editing and filtering are powerful paradigms — but folksonomies (at
least present day ones) suffer from having too little formal structure
– tagging systems easily result in "metadata soup." Ontologies are on
the other end of the spectrum — they are particulary useful for
accurately modeling the actual structure of the world, or of conceptual
domains — but admittedly in some cases their formal structure can be
overly rigid and specific. The benefit of tagging is primarily the
adaptive nature of the resulting taxonomies. The benefit of ontologies
is the rich, and unambiguous, semantics they define. Tagging systems
are useful when all that is needed is the ability to link items to
topics; ontologies are useful when what is needed is to rigorously
define or understand what is meant, or not meant, by particular
classes, fields and relationships — something that is essential for
good machine-processing of data.

One point that Clay makes, which I think is very interesting, is his
view that perhaps the world is moving from a graph-theory information
model (ontologies) to a set-theory model (folksonomies) — but in
fact, under the surface this argument falls apart. OWL is nothing other
than a language for enabling extremely sophisticated set-theoretic
operations on information. In fact, if you actually look at the OWL
language itself, it is primarily comprised of set-theoretic statements.
I don’t really view graph-theory and set-theory as mutually exclusive
– in fact, they are highly connected, if not equivalent at a deep
level. But expressing information in graph form or set form does have
different benefits for certain types of information processing. In
particular, graphs can be beneficial when associative reasoning is
important — for example, when traversing links or networks between
nodes is key. Sets on the other hand are useful when relevance or
mutual membership are most important.

Clay discounts ontologies for many reasons. He has many arguments,
most of which have some merit, but fall short of convincing me (or
anyone in the field of knowledge representation). Indeed, tagging
systems are just special, highly simplistic cases of ontologies –
namely, they are ontologies with extremely basic semantics and almost
no constraints — they are even lower on the spectrum than taxonomies.
In fact, we could graph the spectrum of knowledge management as follows:

<—————————————————————>

Tags Folders Taxonomies Databases Ontologies

One of Clay’s early arguments against ontologies was that they are
merely systems for syllogistic logic — but in fact, that is simply not
the case. While the formal semantics of OWL doe support logical
inferencing and reasoning, that is not the only value of ontologies. In
fact, I think a much more important benefit of ontologies is simply
that they make the semantics of data structures explicit — which makes
it much easier to both process information, and integrate information
across different applications and representations. Ontologies are, in
my opinion, simply the next evolution of database schemas. Surely, Clay
would not argue that database schemas have no place in the world!

Another way of looking at ontologies and the semantic web is that
they do for the meaning of data what other markup languages have done
for the layout and structure of data. HTML provided a way to markup the
formatting of content. XML provided a way to markup the structure of
content. RDF and OWL provide a way to markup the meaning of
information. This is a logical progression, and it is something that
will really make the Web, desktop and enterprise easier to cope with.
Ontologies are not panaceas — but they are incredibly powerful when
used appropriately. And that is the operative word — they are not for
everything. Indeed, in cases where social tagging is sufficient,
ontologies may simply be overkill. But there are many, many cases where
social tagging simply does not, and cannot, have the semantic rigour
that is needed.

So what’s next? I think that ultimately we will see a synthesis of
these two approaches emerge. Imagine a folksonomy combined with an
ontology — a "folktology." In a folktology, users could
instantly propose or modify ontological classes and properties in the
same manner that they do with tags in tagging systems. The most popular
ontological constructs (the most-instantiated classes, or slots on
classes, for example) would "rise to the top" and self-amplify, while
the less-instantiated ones would "fall to the bottom" over time. In
this way an emergent, self-organizing, and self-pruning ontology could
emerge within a community. Such a system would have the ease and
adaptability of a folksonomy plus the semantic richness and formal
structure of an ontology. I think ultimately a
folktology approach will be better than either
folksonomies or ontolgoies on their own.

A Stock Market for Ideas

Nova — Thu, 02 Dec 2004 16:49:39 +0000

Media Mammon is a new stock market for memes. You can invest play money in words and phrases that are spreading through the media. May the best meme win! See also: A Physics of Ideas.

My "A Physics of Ideas" Manifesto has been Published!

Nova — Mon, 01 Nov 2004 19:15:53 +0000

Change This, a project that helps to promote interesting new ideas so that they get noticed above the noise level of our culture has published my article on “A Physics of Ideas” as one of their featured Manifestos. They use an innovative PDF layout for easier reading, and they also provide a means for readers to provide feedback and even measure the popularity of various Manifestos. I’m happy this paper is getting noticed finally — I do think the ideas within it have potential. Take a look.

A Blog Novel

Nova — Mon, 01 Nov 2004 10:04:15 +0000

Rohit Gupta, a Bombay-based writer, who also reads this blog, is writing a blog-novel. He has come up with an innovative way to promote it — by letting readers choose quotes from his text to “own” — by choosing a quote and linking to his blog-novel from it, he will in return link back to your blog from that quote in his novel. It’s similar to my earlier GoMeme experiments, except in this case his novel is the meme that is spreading via a cooperative linking incentive.

Good idea, Rohit! I choose this quote from your novel:

“The other article, an interesting one, is a 2000-word piece on the history of mathematical heretics known as the Circlesquarers, and the transcendental nature of the number Π.“

Great Article on Psychohistory and Sociophysics — Can We Predict Behavior?

Nova — Thu, 21 Oct 2004 02:59:39 +0000

Great find from Rob Usey at Psydex Corporation: This article is a survey of the emerging field of “sociophysics” which attempts to apply statistical mechanics to predict human social behavior. It’s very cool stuff if you’re interested in social networks, memes, sociology and prediction science. The article discusses recent progress towards Isaac Asimov’s vision for a science of Psychohistory as proposed in his Foundation stories. This relates in many ways to my previous article on “A Physics of Ideas” in which I proposed some elementary ways to measure the trajectories of memes as if they were moving particles in a Newtonian system.

Detailed Analysis of GoMeme 1.0 Results

Nova — Thu, 26 Aug 2004 17:03:50 +0000

Greg Tyrell, a PhD student with a strong interest in bioinformatics, has put together a detailed analysis and report on the GoMeme 1.0 experiment, containing several visualizations and results of the survey. Nice work Greg!

Also in other news, Google has started indexing the results. Currently there are 733 results when searching for sites with original, super-long GUID. There are 867 results when searching for the unique string “To add your blog to this experiment, copy this entire posting to your blog, and fill out the info below, substituting your own information in your posting, where appropriate” which was in the instructions — this number should include sites that did not put the whole GUID in. Technorati, which seems to be working better today, finds 58 sites with the long GUID, and none for the instructions text above. So I guess Google wins so far. But I am glad that Technorati is starting to get their bugs fixed! I noticed that blog stats are starting to be updated again.

I also got an interesting link to another Meme visualization, which although having nothing to do with our experiment as far as I can tell, is a nice concept. It takes forever to build out the full visualization and the tree appears to be almost white on my white background making it hard to see, but still worth a look — Meme Tree

Change This

Nova — Mon, 16 Aug 2004 18:27:18 +0000

I am helping Change This, a project to spread manifestos on new ideas by key thinkers. They have asked me to help host one of their manifestos, Creating Customer Evangelists. You can also download it directly from

WASHINGTON, DC—In the interest of national security, President Bush has been asked to stop posting entries on his three-month-old personal web log, acting CIA director John E. McLaughlin said Monday.

According to McLaughlin, several recent entries on PrezGeorgeW. typepad.com have compromised military operations, while other posts may have seriously undercut the PR efforts of White House press secretary Scott McClellan.

A July 24 posting read, “Just got back from a lunch with Colin and Adil Moussa (one of Prince Saud al-Faisal’s guys). Colin wants the Saudis to send some troops to Najaf—so some of the soldiers are Arab, I guess. This Moussa guy sure wears a lot of jewelry. A golden chain, a golden ring with his initials or something, and some other sparkling stuff—kinda effeminate. Anyway, best of luck in Iraq, Iyad.”

McLaughlin, normally hesitant to express public disapproval of the president, said the blog was “ill-advised.”

“I would hate for the president to inadvertently put American soldiers at risk,” McLaughlin said. “We work hard to maintain the integrity of state secrets. When we see the president posting details of troop movements, international counter-terrorism negotiations, and even the nuclear launch codes, as he did on Monday, we have to step up and say something.”

Bush said he could not understand McLaughlin’s anger, characterizing his blog as a “personal thing written for friends and family or whoever” and therefore “none of the CIA’s business.”

Nevertheless, U.S. Secret Service director W. Ralph Basham objected to the blog, as well.

“He is compromising his safety and the safety of those in my department,” Basham said, citing a post from last Thursday in which Bush revealed that he “had to go to some secret meeting with Norquist at some Marriot [sic] over in Virginia.” “Someone could uncover some serious state secrets, if they took the time to wade through all of those photos he posted after he got that digital camera in June.”

On Saturday, Basham asked to pre-screen all blog activity before Bush posts it online.

Bush rejected Basham’s request and later that day wrote in his blog that “Some people who shall remain nameless apparently do not know there is such a thing as free speech in this country.”

Members of Bush’s re-election team have urged the president to exercise caution with his blog, perhaps because of posts like the one dated July 8, 2004: “Another long day of speeches and fundraisers. Met with all these phony media company execs. Had to promise them some bill next term and shake a lot of stupid hands, but they did bring in two or three million or so. Whatever. Karl keeps a list. I got big laughs during my speech, so I’m happy.”

Republican National Committee chairman Ed Gillespie said he spoke to Bush about the blog last month.

“After he mentioned our Monday message-of-the-day in a Saturday post, we’ve really been pushing him to not talk about campaign strategy,” Gillespie said. “He’s not that involved in the planning anyway, so it shouldn’t be too much to ask.”

“We’re not trying to stifle the president’s creativity,” Gillespie added. “We think it’s great he’s taking an interest in writing.”

Bush maintained that he’s doing nothing wrong.

“I know so many people, but I’m way too busy to keep in touch with all of them,” Bush said. “Whether I’m talking about our strategies in Gitmo or my dogs down in Crawford, the blog is an easy way to let everyone know what’s been up with me. If I’ve just had a really good lunch at a new restaurant, or something funny happens in a briefing from the NSA, I want to let my friends and family know about it.”

McLaughlin said it’s likely that Bush will eventually agree to submit his blog for review by the Secret Service.

“Right now, the president insists it’s his right to have it, as long as he doesn’t work on it during White House work hours,” McLaughlin said. “But I believe we’ll be able to convince him, if we let him calm down. And even if we don’t, frankly, I can’t see the blog holding his interest for too long.”

See below for some imporant information…

- – - – - – -
The original article is located at: The Onion

PATH LIST — How This GoMeme Got to My Blog

What is this “Path List” for, what is a GoMeme, and how does adding my blog to the list raise my blog’s Google rankings??? Read the FAQ)

(Instructions: copy this entire article, including this Path List into your blog. Next, add an entry for your blog to the end of the Path List. People who get it from you will then have a link back to your blog. This is an experiment to use a host meme — this funny article — to spread links to blogs. Find out more about the experiment and how it works at the link above. Please join us in testing this!)

1. www.mindingtheplanet.net Minding the Planet, mindingtheplanet309a9
2. (Your Entry Goes Here — add an entry for your blog here in the form: URL, hotlink, optional GUID.)
3. (Add a new line here for the next person after you, containing a copy of the instructions in the previoius line)

GoMeme 2.0 – Help Test This Meme

Nova — Wed, 04 Aug 2004 14:11:31 +0000

Note: This experiment is now finished.

GoMeme 2.0 — Copy This GoMeme From This Line to The End of this article, and paste into your blog. Then follow the instructions below to fill it out for your site.

Steal This Post!!!! This is a GoMeme– a new way to spread an idea along social networks. This is the second generation meme in our experiment in spreading ideas. To find out what a GoMeme is, and how this experiment works, or just to see how this GoMeme is growing and discuss it with others, visit the Root Posting and FAQ for this GoMeme at www.mindingtheplanet.net .

By adding this GoMeme to your Weblog you can get higher Google rankings for your site, and help your friends get higher Google rankings too. Your blog will be linked from every other blog that discovers this GoMeme downstream from your blog (from your readers, their readers, and so on). And that will raise your Google rankings in proportion to the number of downstream bloggers that get this GoMeme from you and post it to their blogs. The more people who blog the GoMeme from your blog, the better your Google rankings will get.

By hosting this meme on your blog, you will also be participating in an experiment to generate a distributed Blog survey and test how memes spread through social networks. The dataset from this experiment is public, open and decentralized — every blog that participates hosts their own data about their own blog. Anyone can then get the whole dataset by just searching Google for this unique string: 98818912959q This code is the “global unique identifier,” or GUID for this GoMeme — it marks every web page that participates in this GoMeme so that it can later be found with all the others. (Note it may take a week or longer before Google indexes your blog, so be patient).

Disclaimer

This is purely an experiment and is just for fun. We are really just curious to see what will happen and this is not a commercial project. Participation is voluntary. We don’t mean to annoy anyone. However, if you don’t have much curiosity, or at least a sense of humor, you may find this experiment to be upsetting. In that case, you might try drinking a good strong cup of coffee. If after that you are still unhappy with us, just don’t read any further and have a great day! (If you don’t want your blog to get better Google rankings, that’s purely your choice!) On the other hand, if you are interested in exploring new technologies and pushing the envelope, then keep reading and we look forward to your participation in this experiment. We also request that participants in this experiment refrain from spamming anyone with this GoMeme. To spread it, just put it on your blog; that should be enough.

INSTRUCTIONS FOR ADDING THIS GOMEME TO YOUR OWN SITE

Step 1 First, to add your site to this experiment, copy the GoMeme to your site from the “Copy This GoMeme From Here” heading above to the End of this article. Please copy this whole article and try not to alter the text so that it is authentic for the people who get it from your blog. If you would like to come up with your own catchy headline, or anything else to help your GoMeme spread, feel free! Let’s see what works best.

Step 2: Now, fill in your answers to these Required Survey Fields (Note: Replace the answers below with your own answers). These will later be automatically data-mined by bots to compile the survey results.

(1) I found this GoMeme at URL: http://www.mindingtheplanet.com

(2) I found this GoMeme on date (day/month/year):03/08/04

(3) I found this GoMeme at time (in GMT format): 01:00:00

(4) I foundit via “Newsreader Software” or “Browsing the Web” or “Searching the Web” or “An E-Mail Message”: Browsing the Web

(5) I posted this GoMeme at my URL (use a hyperlink): Minding the Planet

(6) I posted this on date (day/month/year): 03/08/04

(7) I posted this at time (in GMT format): 01:00:00

(8) My posting location is (city, state, country): San Francisco, California, USA

Step 3: If you’re feeling very altruistic today, also fill in these optional survery fields (Replace the answers below with your own answers):

(9) My Weblog is hosted by: Typepad

(10) My age is: 35

(11) My gender is: Male

(12) My occupation is: Internet Entrepreneur, Ontologist, Writer

(13) I use the following RSS/Atom reader software: Typepad, FeedDemon

(14) I use the following software to post to my blog: Typepad, Zempt

(15) I have been blogging since (day, month, year): 05/08/03

(16) My web browser is: IE

(17) My operating system is: Windows XP

Step 4:Now add an entry for your site after the last entry in the PATH LIST below:
Your entry should be of the form: line number, URL, hyperlink, optional personal GUID for your blog.

(Note: If you would like to track all postings of the Meme that result from your posting of it, once Google has indexed them, you may add your own optional GUID after your hyperlink on your line of the Path List — just make sure it is short, unique, and doesn’t return any results on Google — for example “mysitename137a2r28″. Also note, if the path list gets too long, you should still try to include the whole path in your blog — even if you have to put the list on a continuation page rather than the excerpt for your posting — and make sure others copy the whole GoMeme along with your Path List when they get the GoMeme from you — If they don’t copy it, your blog and your upstream blogs won’t be linked from their blogs).

PATH LIST

1. http://www.mindingtheplanet.net Minding The Planet, mindingtheplanet14798
2. (your Path List entry goes here in the form URL, hyperlink, GUID. Also, please add a new line after this one, for the next person.)

The End

You did it! Now spread it! If all goes well and others find this GoMeme from your blog, you should see some interesting results. Please comment back on the original post and tell us how you’re doing or what you observe, if anything noteworthy happens.

Can You Imagine What Would Happen if MoveOn.Org Used the GoMeme Concept?

Nova — Wed, 04 Aug 2004 09:29:10 +0000

I wonder if anyone from MoveOn.Org or the Republicans will notice our GoMeme experiments? (Not that I’m taking sides — I’ll simply be happy if somebody wins the election!) Grassroots political campaigns could potentially really benefit from the techniques we’re testing here. For example, imagine a “blog meme” for a political campaign — a meme that states some useful facts about a candidate and their opponent, perhaps has some survey questions and a GUID, and has the added benefit of a cool Improve-Your-Google-Ranking-By-Hosting-This-Meme candy coating? Wow — it could spread the message to a lot of blogs pretty quickly if done right. That might actually work. But I try to stay out of politics, so I’m not taking sides here or endorsing anyone. If you read this and know the “right people” — feel free to suggest the idea to them.

FAQ for GoMeme 2.0

Nova — Wed, 04 Aug 2004 06:59:21 +0000

This posting is the FAQ and introduction for a new, improved, second-generation meme experiment that is designed to spread faster and more broadly than the first meme experiment. We call this kind of meme a “GoMeme” (pronounced Go-Meem), because it is a meme that is designed to Go. The actual GoMeme, which you can add to your Website is located, here. Before you do this, please read this FAQ so you know how it works.

Introduction

This new GoMeme is better than the first design because it’s more beneficial to you to participate. Why? Because by participating in this GoMeme, you may be able to raise the Google rank and visibility of your blog. In other words, this GoMeme rewards your blog for hosting it.

In a nutshell, by following the instructions below, your blog will be linked from every other blog that discovers this GoMeme downstream from your blog. And that will raise your Google rankings.

How well does it work? Well, in the first experiment, www.mindingtheplanet.net, made it from not being in the list, all the way up to the number 3 position in the www.daypop.com Top 40 blogs index in just 2 days, even beating Matt Drudge, the New York Times, John Kerry, numerous cool movie trailers, and many other hot topics of the day. Now, in this new experiment, everyone can benefit in a similar way, by leveraging the power of their social networks. For instructions, keep reading…

This new version is also shorter and easier to participate in. For starters, you don’t have to copy all of this text into your blog, you just need the text from the “Copy This GoMeme From This Line” heading below to the end.

Is this Good or Evil?

Before we go much further, let’s clear up one burning question you might have: Is this evil? No, in fact it’s good, it’s altruistic and grassroots — you are helping to promote your friends’ weblogs, while pioneering the use of memes for marketing and pushing the envelope of how Weblogs and social networks can be used. Is this GoMeme a chain-letter? No, it is not a chain-letter, although it may seem like one at first glance. Chain letters are mailed to people, usually without the recipients asking for them. This GoMeme is voluntarily posted by you on your weblog and then voluntarily read and voluntarily spread by your visitors. It is really a vector for an idea — and for your weblog — just like any Weblog posting in fact. (Note: We do not condone chain-letters!)

What is a GoMeme?

So what is this GoMeme thing then? It is a new way to help your friends and get helped by them. It is a new way to grow your blogroll (the list of blogs that you link to from your blog) and a new way to get your blog listed in the blogrolls of other blogs in your social network. By participating in this GoMeme, you, and everyone in your social network, can then get higher Google rankings. Think of it as a much smarter social-network-based blogrolling community. You could actually use this beyond just Weblogs — you could use it to promote any site, or anything in fact. But Weblogs are the perfect medium because they are a form of “social software” — they naturally spread ideas, that’s the whole point of Weblogs.

You may also wonder, what is the actual “meme” being spread by this GoMeme? Well, looking at it one way, it is this entire posting. Looking at it another way, there are many memes within this posting, such as the concept of a “GoMeme” and the many other ideas here, the concept of growing a blogroll via a social network, and the concept of a blog, all of which are memes in their own right — and your weblog URL could soon be one of these memes, if you participate.

A Collaborative, Distributed, Emergent Blogroll

This GoMeme is effectively a collaborative, distributed, emergent blogroll. It is no different really than any blogroll (any list of other blogs you add to your blog) — it just forms in a different way. Instead of you adding all the links to it, your social network adds them and then you add yours at the end and send it on to others in your social network. There’s no top-down control or guidance of the process. Every blog that participates is equal. Nobody knows what the result of this experiment will be.

Why Are We Doing This?

OK, so why are we doing this? The short answer is, “Because we can!!!” But seriously, we’re also doing this because it is an interesting way to generate a dataset that we and others may study to analyze how ideas move across social relationships on the Web, and how communities can emerge and self-organize. It’s fun and it’s research, and anyway, it’s harmless and it helps bloggers get better visibility, so we figured we’d just try it and just see what happens! We hope you’ll join us!

How Does it Work?

Just follow the instructions to fill out your blog’s answers to the GoMeme survey and add your blog’s URL to the “PATH LIST” at the bottom of the GoMeme. The path list is the history of all the blogs that the meme traveled through to reach you. The last URL before yours in the path list should be the URL for the blog you discovered the GoMeme on. By adding your URL after it, your blog URL becomes part of the path for the GoMeme.

Everyone who gets the GoMeme downstream from you will then include your URL on their blog. And by doing that, they are in effect linking to your blog from their blog, which in turn raises your blog’s Google rank. By posting this GoMeme to your blog, you help raise the rankings of every blog in the path before yours, and every blog that later posts it as a result of your blog is then helping to raise your Google ranking. Kinda cool, huh?

Can Your Blog Out-Rank the A-List Blogs?

If the new GoMeme design works as we hypothesize it might, it could help a lot of lesser known blogs get better rankings than even the “A-List” blogs. In other words, it’s kind of like unionizing to beat the big guys. There is strength in numbers, after all. Let’s see if this works! If it does work, we should find lots of lesser-known blogs that participate in this experiment appearing in the “top lists” of the major blog indexes. Who knows, maybe we can even take over the top lists? For some proof, just look at the results of the first meme experiment which launched the Minding The Planet weblog to the number 3 slot of the Daypop Top 40 blog posting index in just two days! Who knows, maybe it will be your blog up there next?

OK, so do you want to add your Weblog to the GoMeme experiment??? Great, then Get the GoMeme!

RFC for a New Distributed Data Exchange System

Nova — Tue, 03 Aug 2004 00:11:52 +0000

Matt Poepping has come up with an interesting idea for how to create a fully distributed searchable database on the Net. It’s a cool enough idea and approach that people should see his RFC and comment on it. He may be onto something important here.

A New Blogging Feature: Automated "Social Syndication" Networks

Nova — Mon, 02 Aug 2004 21:35:23 +0000

Here’s an idea I’ve had recently that is related to the Meme Propagation experiment (see posts below on this blog for more about that ongoing experiment). The concept is for a new, meme-based, way to syndicate content across blogs. Here’s how it might work:

1. You join a “meme syndication network” by joining at a central site. You get an account where you can profile your blog. You also set your blog’s syndication inputs — a set of other blogs that are also in the network that you are willing to automatically syndicate content from.

2. When you complete this, you are given an automatically generated HTML element containing a script to put in your blog sidebar, or anywhere else in your layout. This script is auto-generated for you from a central site that manages the network. The script automatically displays short excerpts for blog postings (pieces of microcontent) that have been “picked up” by your site from your registered “inputs” in the network. You place this script in your layout.

3. In the area created by the script in your site, you see a listing of blog postings that have been syndicated to your site from your inputs. You can post to your network by going to your account at the central network site and posting (or copying in the URL for anything you want to post) there. Any network-member sites that treat your node in the network as an “input” will then *automatically* pickup your posting and display it on their page.

Using this method, you might for example, post an article and suddenly have it appear syndicated across hundreds of sites that are downstream from you in the network. Your posting would cascade from node to node, via the network of nodes that directly or indirectly treat your node as an input.

Postings to the network would last for a certain number of hours or days and then expire. Furthermore, each member’s posting list would only have room to display perhaps 20 headlines in a list form, and preference would be given to postings that are more recent and from nodes that are closer to them in the network. Perhaps postings that were more popular could somehow last longer too, and get higher preference in listings. In this way the network would act as a natural filter.

Now, let’s just add one more feature. The postings that are syndicated in this manner could be similar to the Meme in the Meme Propagation experiment. Meaning that they could have an unique GUID on them so that you could easily find all places where the meme occurs. Better, yet, they could be actual objects — so the actual posting is syndicated, rather than a copy of it. Thus when people comment on it at different blogs, they are all really commenting on the same underlying data object — so all comments and activity around the posting are part of the original posting (this is in contrast to the way many blogs do citations — where each citation to a posting is really a new posting and comments that are added to it go directly onto it rather than onto the original posting, which causes fragmentation of the conversation).

So this concept provides for a means to syndicate memes across a social network of blogs that opt-in to auto-syndicate content from one another. A meme posted to your node could quickly appear across numerous downstream nodes in your network — effectively increasing your footprint for that posting. The central site could then track the statistics of memes in the network, administer the lifespans of memes, and provide useful overviews to the community.

Anyone want to code this? Let me know. I want to use it!

Is Your Blog a Hot Zone?

Nova — Mon, 02 Aug 2004 20:41:56 +0000

Meme Update: The Meme is already global and the rate of growth is showing signs of exponential increase. It’s made the Daypop top list, also same with Blogdex. It’s made its way onto several early-adopter sites and lists. Already the results are interesting. One thing that is clear is that there is quite a lag time in Blogspace: This applies not just to blogs, but also to aggregation sites and search sites — which don’t update nearly as often as one might think.

Comments:

It seems that certain bloggers read and post much more frequently than others — we could call their blogs “hot zones,” to borrow a term from epidemiology.
.

The hot zone blogs are the blogs that are most interesting to read — they have their fingers on the pulse so to speak — they are where memes are born and vectored first. What’s interesting is that these hot zone blogs are not necessarily the most well-known or most popular blogs. In fact, I am surprised to see that some of the more well-known blogs have not got the Meme yet! So this is interesting — the blogs that are most on top of current trends are not necessarily those that are the most popular.

From the hot zone blogs, next we should expect to see “second wave” blogs pick up the trend, which is how it will go mainstream. The hot zones are the key though — they are the blogs we should all be reading if we are interested in tracking early-stage trends, before they have become mainstream. How do we define a “hot zone” blog? It should be a blog that picks up memes early, but also has a significant impact on the spread of the meme — a blog that results in lots of other blogs posting the meme.

Once this experimental data set gets large enough we should be able to compute, for each blog that participated, the number of downstream blogs that directly and indirectly got the meme from it. From this we should be able to compute exactly which blogs are the key “hot zones” in the process

GoMeme 1.0 — Testing Meme Propagation In Blogspace: Add Your Blog!

Nova — Mon, 02 Aug 2004 01:22:01 +0000

NOTE: This experiment is now finished.

This is an experiment in spreading ideas across weblogs using the principles of viral marketing and social networks using a new method for making content more viral, which we call a "GoMeme."

How To Participate

We have created several versions of this meme in the course of the experiment. The latest version is GoMeme version 4.0. It is the most refined version; You should try this one if you are interested in participating.

Basically we are experimenting to see which kinds of conceptual motivations spread memes best. The first test was a survey which was purely altruistic. The second was survey with a selfish motivation — a path list so the participant benefits. The third used a humorous article as the host for the meme. The last one in the experiment used a "cause" as the host of the meme. Which one will be the most viral? It’s also an experiment in the concept of a path list — which helps blog authors see the extended upstream and downstream paths for things they post or find on their blogs.

The previous versions of the experiement are listed below for your reference:

- GoMeme 1.0 — Our first test. (The text for GoMeme 1.0 is located below, in this article, after the "Introduction to Gomeme 1.0" heading. This first test spread very rapidly but did not benefit the parties that helped spread it in any particular way.

- GoMeme 2.0 — A more evolved version. It has the benefit of not only spreading virally but also of rewarding every site that helps spread it with better Google rankings.

- GoMeme 3.0 — This is the most advanced version, and also the simplest and shortest. We abstracted the essential core of what makes a GoMeme spread and made it possible to attach it to any piece of content on the Web. So now anything can spread in this manner.

- GoMeme 4.0. This version is the latest and most refined of all. It is a very short posting that you can add to your blog. Unlike the earlier versions, it is focused around a cause — the cause of making blogs more visible! Let’s see if having a cause attached to a meme makes it spread more widely.

- – - – - – - – - – - – - – - – - – - – - – - – - – - – - – - – - – - – - – -

Note: The information below is the first version of our GoMeme experiment (GoMeme 1.0) — it has been replaced by more recent versions that are more sophisticated and that also reward every site that participates with higher Google rankings.

If you would like to test this out, we suggest you try either – GoMeme 2.0 or
- GoMeme 3.0

You may also want to read the GoMeme 3.0 FAQ

The most recent version is: GoMeme 4.0

The rest of this article, below, is the body text of
the original GoMeme 1.0. This is reproduced for informational purposes
only. The experiment is finished, please do not continue it.

———————————————————————————————————
GoMeme 1.0

This posting is a community experiment that tests how a meme, represented by this blog posting, spreads across blogspace, physical space and time. It will help to show how ideas travel across blogs in space and time and how blogs are connected. It may also help to show which blogs (and aggregation sites) are most influential in the propagation of memes. The dataset from this experiment will be public, and can be located via Google (or Technorati) by doing a search for the GUID for this meme (below).

The original posting for this experiment is located at: Minding the Planet (Permalink: http://novaspivack.typepad.com/nova_spivacks_weblog/2004/08/a_sonar_ping_of.html ) — results and comments about the experiment appear at that location. (Note: If you want to replace this text version for the link for better formatting in your blog, please do at least include a hotlink back to the original post so that we can see your trackback).

Please join the test by adding your blog (see instructions, below) and inviting your friends to participate — the more the better. The data from this test will be public and open; others may use it to visualize and study the connectedness of blogspace and the propagation of memes across blogs.

The GUID for this experiment is: as098398298250swg9e (Note: this replaces the longer, original GUID — listed below — which didn’t format nicely in narrow column layouts. Those sites still using the longer GUID will still be found in the data set).

The above GUID enables anyone to easily search Google or other search engines for all blogs that participate in this experiment, once they have indexed the sites that participate, which may take several days or weeks. To locate the full data set, just search for the any sites that contain either the short GUID (above).

Anyone is free to analyze the data of this experiment. Please publicize your analysis of the data, and/or any comments by adding comments onto the original post (see URL above). (Note: it would be interesting to see a geographic map or a temporal animation, as well as a social network map of the propagation of this meme.)

To participate, follow the instructions, below…

INSTRUCTIONS

To add your blog to this experiment, copy this entire posting to your blog, and then answer the questions below, substituting your own information, below, where appropriate. Other than answering the questions below, please do not alter the information, layout or format of this post in order to preserve the integrity of the data in this experiment (this will make it easier for searchers and automated bots to find and analyze the results later).

REQUIRED FIELDS (Note: Replace the answers below with your own answers)

(1) I found this experiment at URL: http://www.mindingtheplanet.com

(2) I found it via "Newsreader Software" or "Browsing the Web" or "Searching the Web" or "An E-Mail Message": Browsing the Web

(3) I posted this experiment at URL: http://www.mindingtheplanet.com

(4) I posted this on date (day/month/year): 01/08/04

(5) I posted this at time (24 hour time): 17:50:00

(6) My posting location is (city, state, country): San Francisco, California, USA

OPTIONAL SURVEY FIELDS (Replace the answers below with your own answers):

(7) My blog is hosted by: Typepad

(8) My age is: 35

(9) My gender is: Male

(10) My occupation is: Internet Entrepreneur, Ontologist, Writer

(11) I use the following RSS/Atom reader software: Typepad, FeedDemon

(12) I use the following software to post to my blog: Typepad, Zempt

(13) I have been blogging since (day, month, year): 05/08/03

(14) My web browser is: IE

(15) My operating system is: Windows XP

An Interesting Visualization of Word Frequencies

Nova — Thu, 29 Jul 2004 04:40:18 +0000

This animated visualizer lets you enter a word (in the little search box on the bottom left) and then shows the word situated next to other words that are used with similar frequency in English. It’s cool — you can discover some interesting things. Read the about page for more on that. This system would be really good if it used the concepts from my paper on The Hacker’s Dictionary)

Memes are essential to the way the human brain processes ideas and how it decides what is important. We are basically “meme processors” — we are “life-support systems for memes” to put it another way. To use a computer analogy, our physical bodies are like the hardware and operating system, and our minds — the dynamical activity and state of this hardware — are like the software applications and content running on the hardware. Our minds could be viewed as systems of interacting memes — complex systems of ideas that interact within us, and across our relationships.

Memes are capable of spreading across human social relationships via human interactions, and via human usage of static storage vehicles such as printed media, audio or video, and digital storage media — they are highly “communicable.” (And soon, as I have proposed in other articles, with the coming Semantic Web memes will be able to spread and interact without needing humans at all — machines will be able to process them on their own).

The Media is the Mirror

Before we can measure the physical properties of memes, we need a way to identify the memes we are interested in analyzing. We can identify memes by analyzing textual media such as document collections, wire services, and the Web.

The memes within text appear to be dormant — they are frozen digital representations. They do not move or reproduce on their own — they need help from humans (for the moment). But by inference, static textual representations of memes provide a mirror of the actual “active memes” that are taking place in the minds of the people who author and consume that media. What this indicates is that by analyzing textual media we are not merely looking at the memetic properties of text, we are looking at the memetic properties of people’s minds and of organizations, societies and cultures. In a sense, by selectively choosing the right media we can make a virtual focus group — we can see what people in this group are thinking.

The media is a mirror of our minds and cultures. By analyzing suitably selected information sources (for example, “all news articles from USA newspapers”) we can effectively focus on a reflection of the memes that are actually present within the minds of humans in a particular place, time, industry, community, demographic, etc. The more we know about the information sources, the more we can infer about the memes we find, and thus the memes taking place within the minds of the people who interact with those information sources.

The simplest approach to identifying memes in textual media is to simply pre-specify a list of memes we are interested in and to then search for any matching strings. For example we might be interested in measuring memes related to a particular trend, such as “Java technology,” so we could compile a list of terms related to Java and then use search techniques to locate all instances of those terms. We can then measure their properties.

A more sophisticated approach than specifying interesting memes in advance is to discover them empirically by analyzing text to see what’s there. To do this we might automatically identify nouns or noun-phrases and then measure their dynamics to see whether they are interesting enough to warrant further analysis. There are many existing computational liguistics techniques for isolating parts of speech and linguistic expressions.

Each of these nouns or phrases is a potential meme (we may consider them to all be actual memes or we may filter for only those memes that exhibit dynamics in space and time that meet our threshold for what constitutes “interesting” or “memelike” behavior. Another, more brute-force approach, would be to simply analyze every noun and phrase in a document or corpus for any that exhibit “memelike” dynamics in order to discover memes empirically instead of specifying them and then gathering their stats.

We can use various standard methods from text-mining and natural language processing to do a smarter job of identifying memes (for example, we can use stemming to consolidate various forms of the same word, we can use translation to consolidate expressions of the same meme in different languages, and we can use conceptual clustering and even ontologies to consolidate different memes that are equivalent to the same underlying meme). But for now, we can start by identifying memes in a simple way — the same way we might identify “topics” or “keywords” in a document. Once we can do this we can then measure the physical properties of those memes as they move through time and various spaces of interest.

(Note: We don’t necessarily have to analyze every document in a corpus to gather valid statistics for memes within it. We can use random sampling techniques for arbitrary degrees of accuracy if we wish to optimize for faster results and less computation. Instead of analyzing every occurance of each meme, we can analyze a statisically valid sample of the corpus.)

The Physics of Ideas

I suggest that the physics of ideas will be quite similar, if not equivalent to, the physics of the natural world. Everything in the universe emerges from the same underlying laws, even memes. The intellectual processes taking place within our own minds, as well as across our relationships and social organizations are similar to the dynamics of particle systems, fluid flows, gasses, and galaxies. We should therefore be able to map existing physical knowledge to the memescape, the dimension of memes.

Here are a set of basic measurements of the physical properties of memes and documents:

(Author’s Note, February 28, 2010: My latest thinking on this topic has evolved considerably from when this article was originally written in 2005. Instead of viewing memes as classical particles, I now think it is probably more accurate and useful to model them as physical waves or fields. At any given location (a media outlet, or a geographic place, or a person or document) every meme can be represented as a vector at any given time. In any case, regardless of the particular physical model we choose to map to memetics, the key point here is that it should be possible to make such a mapping from physics to memetics. This is a testable hypothesis. For example, select a certain mapping and generate some measurements about the higher order dynamics of memes, and then see if we can make testable predictions from those. Through such a process it should be possible to experimentally test and verify whatever mapping we choose, to find mappings that are most useful and accurate. Once we choose a mapping from physics to memetics that works, it could be an extraordinarily powerful tool for making sense of what is going on in the world, and particularly on the Web. I leave it to the physicists among us to come up the correct model, mappings, and experiments. In addition, since the original date of publication, social media has become an enormous playing field for memes and particularly rich source of data for measuring and mapping meme dynamics. In addition to documents we may also think of people and their lifestreams as sources of memetic data for measuring memes. Below is the original proposed mapping — which primarily was a classical physical model, focused on documents only.)

- Absolute meme mass = how “large” the meme is. There are various ways to come up with a measure of mass for memes and I don’t claim to have come up with the only, or even the best, way to do so. This is still a subject for further investigation. However, to begin, one approach at least is to interpret the mass as the total number of times a meme is mentioned in the corpus since the beginning of time to the present. However, it has been pointed out that this interpretation will cause the mass to increase over time. Still, it may be a useful interpretation, and in this paper I will use it provisionally. Another and perhaps better possibility, is to quantify the relative importance of particular memes in advance (for example by having analysts rate the terms that are most important to them) and to use these values as the mass of those memes. Note: When computing meme mass, we can choose to count repeat mentions or ignore them — doing so has slightly different effects on the algorithm. We can also, if we wish, get more fancy and look at clusters of memes (via semantic network indexing or entity extraction, for example) that relate to the same concepts in order to compute “concept-cluster momenta” but that is not required.

- Absolute meme velocity = how fast the meme is moving in the corpus in the present time interval = The rate of occurrences (or “mentions”) of the meme per unit time (minutes, hours, days, etc.) in a given time interval.

- Absolute meme momentum = the force or importance of the meme in the corpus = the meme’s absolute mass x the meme’s absolute velocity

- Relative meme mass = the mass of a meme within a subset of documents or data in the corpus representing some set of interests. (Note: we call a subset of mutually co-relevant documents a “reference frame” or a “context.”) such as a set of interests, a particular period in time, etc. (rather than in the entire corpus).

- Relative meme velocity = the velocity of a meme within a reference frame.

- Relative meme momentum = the relative meme mass X the relative meme velocity.

On the basis of these we can then compute derivatives such as:

- Absolute meme acceleration = how the absolute meme velocity is changing in the entire corpus = The change in absolute velocity per unit time of the meme in the corpus.

- Relative meme acceleration = the change in relative velocity of a meme.

- Absolute meme impulse = the change in importance per unit time = the change in a meme’s absolute momentum.

- Relative meme impulse = the change of a meme’s relative momentum.

Next, we use the above concepts to look at sets of memes, for example documents:

- Absolute document momentum = the force or importance of a document in the entire corpus = the sum of the absolute momenta of each meme that occurs in the document. (Note: we may choose to count or ignore repeat occurrences of an article in different locations or at different times — this has different effects).

- Relative document momentum = the force or importance of a document within a reference frame = the sum of the relative meme momenta in the document. This is a more contextually sensitive measure of document momentum — it couples momentum more tightly with a context, such as a particular query or time interval, or demographic segment. (Note: we may choose to count or ignore repeat occurrences of an article in different locations or at different times — this has different effects).

- Hybrid document momentum = a measure of momentum that combines both relative and absolute measurements = either relative mass X absolute velocity or absolute mass X relative velocity.

How To Analyze a Corpus Using These Methods

We can then apply the above measurements to entire corpora (collections of documents). This enables us to empirically rank the ideas occurring in the corpus in any interval of time. Furthermore it enables us to rank and prioritize documents in the corpus according to their momenta within any time interval — in other words, how representative they are of “important” or “timely” ideas within any time interval.

To do this, first we must create an index of stats for all memes we are interested in. We can use the above mentioned techniques for identifying memes to do this. For each meme we identify, we create a record in our index that lists the stats we find for it by source location and time. We then analyze our text sources and update the records in this table (for a historical analysis we do this all at once; for a real-time analysis we do it continuously on an ongoing basis or in batches). As new instances of memes are found we append the corresponding records in the index.

We can now use these statistics to plot memes and documents according to our measurements of meme and document mass and velocity. This enables us to segment the memes or documents according to the various possible configurations of these dimensions. Each of these configurations has a useful meaning, for example a document with low absolute mass, moderate or high relative mass, high absolute velocity and high relative velocity contains “newly emerging trends of interest to the current context” whereas a document with high absolute mass, low relative mass, high absolute velocity and low relative velocity contains “established large trends that are not very relevant to the current context.”

By looking at the impulse (the change in momentum) we can also chart the direction of these trends (increasing or decreasing). Memes that have high positive impulse are becoming more “important” than those with lower impulses. This enables us to determine whether memes are “heating up” or “cooling off” — a meme is heating up if it is important and timely and has positive impulse.

Thus documents that have high document momenta contain memes that have high meme momenta — in other words they are representative of whatever ideas happen to be most important now. Tomorrow, when the momenta of various memes may have changed, the same documents might have different document momenta.

These techniques provide a way to rank documents that is in some respects like Google’s algorithm, except that it works for all types of information — not just information that is highly interlinked with hotlinks or citations but even for flat text — and it is capable of arbitrary resolution in time and space. For example, Google is basically estimating document popularity — or effectively, endorsements implied by citations — for each query. Google determines the rank of a page in a set of results by estimating the community endorsement of that page as implied by the number of relevant pages that link to it. Using the proposed physics of ideas however we can accomplish the same thing in a different and possibly better way — we can now compute the ‘potential community value’ of a document — without actually requiring links in order to figure that out. Instead, we can determine the relative strength of the ideas (the memes) that are present in the document and compare them to the memes that are present in the community of documents that are relevant to the keywords in our query.

For example, we do a query for “space tourism” and get back 6,830,000 documents in Google. Next we compute the above stats for each of those documents. We then rank the documents returned by this query according to their relative document momenta. This has the effect of ranking the documents according to the strengths of memes that are particularly of interest to the community represented by the query results. Thus it enables us to rank the resulting documents for our “space tourism” query to favor those documents that contain the highest momentum memes relative to set of memes that matter to the community — in other words the documents that contain ideas that are most “timely for the community” would appear higher. So this is a way to figure out not just what is relevant but what is important or in other words timely at a given point in time to people with a given set of interests.

Example Applications

Using the above techniques we can use momentum to provide a more sensitive way to filter any collection of information objects for which we can gather stats representing mass and velocity. There are numerous useful applications of doing this. Below I describe some of them.

Filtering E-Mail

For example, one might filter their e-mail using meme and document momenta in order to automatically view messages, people and topics with high momentum, low momentum, growing or declining momentum, etc. One could also use these techniques to data-mine the articles in a news feed or corpus for those that contain the “hottest trends.” It could be used to automatically detect “emerging hot topics,” “people to watch,” “companies to watch,” “products or brands to watch” etc. When ever you send a message the system measures the memes in that message and updates a special meme-stats index called “my interests” which just has the meme-stats for memes in messages you send. All incoming e-mail messages you receive can then be ranked according to their document momenta with respect to the meme momenta in the “my interests” index. This e-mail filter is automatically adaptive — as you send messages it learns what your current interest priorities are and this is reflected in changing meme momenta, even as your interests shift over time. These updated momenta are then used to filter incoming mail. So your mail filter learns what is important to you as you work and adapts to focus on your current priorities and interests, without you having to teach it. It just learns and adapts to model your current interests as you work.

Media Analysis

Beyond just that, these techniques can be used to perform more precise media analysis — for example they can be applied to measure the success of an advertising or marketing campaign by correlating the campaign placements with changes in momentum of the memes for the brand or product in the media.

Predicting Changes to a Stock Price

We can also use these techniques to make predictions — for example, we can correlate meme-momenta for memes related to a company with technical properties of its financials and stock price and then make predictions about price changes by analyzing news articles to detect changing meme-momenta related to the company. We can also do pure statistical correlations between meme momentuma and stock momenta for example. The financial news media is like a mirror reflecting what is taking place in the markets — but investors also use this mirror to decide what to do in the markets. So by measuring what appears in this mirror we can predict what investors are likely to do next.

Prioritizing Search Results and Implicit Query Expansion

We can also use these techniques to prioritize Internet search results — or any search results for that matter. For example, a set of Web documents can be prioritized by their document momenta, such that those that represent the memes that are currently the hottest can score higher — in other words, documents that are currently more timely can score higher than those that are less timely, and documents that are more timely yet less relevant (on a keyword level) can be ranked higher than those that are less timely but more keyword-relevant.

For example, suppose you search for “Asian restaurant.” If the meme “Vietnamese food” is currently in vogue in the media, meaning that it has higher momentum currently, then documents about restaurants that contain “Asian” or “restaurant” and that contain “Vietnamese food” will score higher than those that only mention “Asian” or “restaurant’ and “Chinese food” (assuming that Chinese food currently has a lower momentum). But this could change later as trends change. In other words, although we searched for “Asian food” we ended up getting documents ranked not merely by the keywords “Asian food” but by what topics related to Asian food have highest momentum today. This is a form of “implicit query expansion” and “implicit filtering.” In other words the system can prioritize search results for you according to the present momenta or in other words, the timeliness, of memes that occur in them. So it can show you the documents that are likely to be most important to you NOW in light of current trends and events, versus just the documents that have the best keyword relevancy.

Market Research

To make things even more interesting, we can add additional arguments to our “Rank of item” function and our meme-stats table — for example, not just a measure of mentions but also a measure of “hits” — hits on a meme increase whenever a document containing the meme is viewed. We can also add another dimension to represent the spatial distribution of memes. This will enable us to track the vectors of memes through time and space. We can do this by associating each source (each publisher) with a geographic location. We then segment our meme-stats table by geography to break out the momentum of each meme in each geographic region. This enables us to do things like filter documents by “how important they are to people in New York.”

By adding further dimensions — such as demographic profiles gleaned for example from the reader-surveys of publishers we can also segment by demographics, so we can even filter documents by “how important they are in the last month to professional, Democratic party affiliated, college educated, women in New York City who earn a median household income of $100,000.”

By adding still one more dimension to measure “sentiment” for each mention of a meme (as a function of the positive or negative language occurring near it or better yet, about it), we can even start to rank memes according to the percent of members of a given population that support or oppose them.In other words, this system can be used to empirically measure what polls and focus groups do informally. The notion here is that by selecting media sources that are representative of the community you are interested in understanding, you can then view memes and meme data relative to that group. You can also do this in the other direction, simply look to discover what memes have interesting stats for the group your are interested in. Another use of this technology might be to analyze intellectual history by computing meme-stats from historical documents or past news articles.

We can also leverage the fact that meme dynamics can be corellated with those of other memes to determine dynamical dependencies amongst them. This enables us to determine that some memes postively or negatively reinforce others. It also enables us to discover sets of related memes — such that we can learn that stats on a given meme should be inherited by related “child memes” in an automatically or manually generated taxonomy of memes.

Measuring and Mapping Ideas in the Semantic Web

We could also reference metadata about the semantics of various memes we can even filter for various types of memes — such as “memes related to vehicles” or “memes representing people” or “memes representing products ,” etc. This enables us to start measuring ideas as they occur and interact on the emerging Semantic Web — but not just particular memes, even conceptual systems of memes that are interacting or somehow ontologically related. By linking with an ontology, for example, we can track the momentum of all memes related to “American cars” versus those for “German cars.” The ontology enables inferences that help us find all memes that represent types of cars and classify them by nationality of manufacture.

Intelligence Analysis

These techniques might even be used to detect signs of potential terrorism, and to “get inside the minds” of various people or groups of interest — simply analyze the meme-stats for memes in documents they create or view to automatically generate a profile of the main ideas currently occupying their minds. Next by tracking this over time you can start to plot trajectories and make predictions. Intelligent agents can then be trained to notice “interesting” patterns in these trajectories and alert analysts as needed.

Advertising Targeting

The same methods could be used to better target advertisements or recommendations to users. Knowing what memes are currently most important to a party enables better personalization and targeting. In this case a Web site could track what memes are hottest for a given user account — derived from what pages they view and what messages they write or respond to. This data could then be used to augment the users’ interest profile with more dimensions of detail about each interest — such as how timely it is to the user, what particular nuances are specifically interesting, what their sentiment is. This could result in less irrelevance and spam for users and better results for marketers.

Knowledge Discovery

Now what gets interesting is the above methods can be used on both directions. We can use them to ask questions about memes we are interested in and we can also use to empirically discover memes we should be interested in within any corpus. So for example we can just empirically compute meme momenta and document momenta in any collection of information and then filter for whatever dynamics we are interested in, for example, “hot new emerging trends to watch.”

A New Kind of Portal

Using these methods it is possible to build a new kind of portal that provides a window into the collective mind of the planet (or any community of interest). It would show what people within the desired segment think is important over time. We could watch an animation on it of how memes for “Jihad” have spread, or for how those for a technology like “Java” have spread versus those for “Microsoft .Net,” or how a particular war is currently viewed by the public in different states or different demographic segments. A user could “drill down” into any meme to see it’s stats, all articles where it was mentioned, and related items on the Web, and maybe even products etc.

Open Questions & Directions for Further Research

It is important to note that these simple physical concepts could be taken much further. For example, using the above approach we should be able to determine the “gravity of a meme” or of a document or any set of memes or documents. We can then start to model the shape of memetic manifolds — the shape of space-time for ideas. We can also start to look at systems of memes as fields. Perhaps there may even be applications of fluid dynamics, relativity theory, or even quantum mechanics to what is taking place in the memescape — but today we are just taking baby-steps, just as Newton and the early natural philosophers did long before us. We need to begin to simply have the ability to measure memes and their basic interactions before we can go on to higher levels of analysis. I leave it to the physicists among us to take this to the next level of formalism — would anyone like to try their hand at formalizing the above proposed equations for the physics of ideas, or perhaps proposing even better ones?

There are a number of open questions I am still thinking about that suggest opportunities to refine these techniques. In particular, should we normalize documents somehow so that large documents don’t have an unfair advantage over small documents (because large documents have more terms in them and thus have higher document momenta)?

Another question is whether or not we should rank documents first by relevance to query, and then within each “relevancy band” further rank by document momentum within that band? This has the effect of limiting the impact of momentum versus relevancy — which may be useful if relevancy is considered to be more important. For example the top 100 most relevant documents are ranked by relevancy and then within that set they are ranked by document momentum and displayed, next the second 100 most relevant documents are ranked by relevancy and then within that set they are ranked by document momentum and displayed, etc.

Another question is whether there is an ideal set of priorities for the various measurement dimensions above with which to rank documents for general searches. We can let users choose their own priorities of course, for example, by letting users set their priorities for various memetic dimensions, we can then tailor our ranking for their needs. Are they just looking for all documents that are relevant to a query, or are they really trying to find documents that are representative of the most timely issues relevant to a query? We might enable users to set their weights for the absolute and relative measurements of documents in order to view different rankings of search results. Better yet, we could simply provide them with natural language filters to apply, such as “Filter for documents that contain currently hot topics related to this query.” In other words they can set priorities for the above dimensions in order to favor one dimension over another — so they might decide that query relevance is most important, document mass is second and velocity is least important. This would translate to a constraint such that it would be more difficult for documents with low relevance to be ranked higher than documents with high relevancy just because they have higher momenta On the other hand, they might want to favor momenta — for example if they really want to find documents that mention the latest trends related to a query — in which case we would favor document mass and/or velocity above document relevancy in our ranking. I am still thinking about the best way to handle these tradeoffs. Letting the user set their priorities is one way — but it may be possible to do a good job of satisfying most people with a particular set of default priorities. What is the best set of default priorities for general use?

There is also the question of how to best represent the “footprint of a meme” in geographic space. We can detect mentions of memes and using the above methods we may be able to associate each mention with a particular geography (the geographic region of the publisher and/or the intended audience — if the source has an audited audience demographic survey — as most publications that sell advertising do — then it is easy to associate any memes that occur within its content with particular geography and demography). Now the question is suppose we are tracking a particular meme — can we determine its geographic trajectory over time? Can we determine the vector of each meme at each sector in a geographic map? And can we represent that in an animated map for exampe, perhaps with something like a fluid flow animation?

Another open area to study is to analyze the higher order distributions of memes in order to automatically detect memes that are “interesting” (ie. not “noise” according to our priorities). One easy way to do this is to automatically ignore any memes that have a random distribution. We may also want to de-emphasize memes that have regular distributions — such as memes for which the dynamics have been the same for a reasonanble period of time. In other words, we want to filter for memes that have dynamics which deviate from being predictable or stable (randomness and regularity are both predictable). My hypothesis is that the really interesting memes — the memes that represent important emerging trends or current hot issues — will exhibit high volatility. For example, imagine for a moment that we are tracking memes related to “digital music” — if we look back in time there will be a point where the word “Napster” suddenly appears — at first it is a relatively “small” meme but gradually it spreads and gains momentum. Then there is a critical point where it begins to grow exponentially. Then it probably levels off for a while or even inflects after the initial hype phase ends. Next another dramatic increase in momentum should be seen around the time of the music industry’s lawsuits against Napster. Then following the resolution of these we should see Napster fall off dramatically. Later we see momentum increase again as the new commercial version of Napster is announced. This type of pattern is what we are looking for. Can we characterize these patterns well enough that we can detect them automatically?

Perhaps one way to do this is by training a neural network to recognize the types of patterns that interest us — we could do this for example by taking historical content (such as the last 10 years of the Associate Press) and then telling a neural net what memes are most important to us. The neural net can then learn from this training data. We can then run the neural net on current or more recent news and let it guess what is important to us based on the patterns of past important trends. We can rate these guesses to provide further feedback to improve learning. This approach could be used to train intelligent agents that specialize in detecting particular types of trends — for example, we could train agents to alert us when a major new technology trend is about to erupt, or when we should invest in a technology stock, or when a company we track is experiencing a major change of some sort, or to tells us when a new competing product emerges or when an existing competing product overtakes our own product, etc. We could also potentially train agents to recognize the early signs of important cultural or political issues, significant changes in sentiment or focus for a given community we are interested in, or even signs of emerging threats.

Are There Ideal Meme Distributions?

Perhaps one of the most interesting questions I have thought about in relation to the physics of ideas is whether or not there are perhaps “ideal distributions” of memes that get the best response from humans? In other words, do the higher order distributions of memes that become major trends, or that get the most attention in noisy environments, have similar characteristics? If it turns out that this is the case then it could provide a powerful new technique for advertising, information filtering, and even for user-interface design. I believe we can analyze memes to answer this question. Here’s how we might do it:

Approach 1: We choose a representative set of memes for major trends. We analyze their higher order distributions in the media. We then attempt to figure out whether these distributions have anything in common that we can isolate. We then search the media for other memes that have distributions with similar properties and test whether they are in fact major trends. We can provide feedback by scoring the output of these trials and using an evolutionary algorithm to evolve successively better filters. Eventually through such a process we can evolve an agent that is good at discovering major trends in the media.

Approach 2: We can do a perceptual psychology experiment to discover and evolve memes that get the most attention. Create a noisy environment in any sensory modality — let’s use visual information for the moment. Put 100 human subjects in a room and show them a computer generated slideshow. Our slideshow consists of 100 images. We change slides rapidly. Each slide is shown many times in the course of the slideshow, with a frequency according to one of many different distributions we wish to test. For example, one slide is shown such that it has low mass, low velocity — a low momentum. Another is shown to have high momentum. Others are shown to vary such that their momentum inflects and is volatile. We can test a number of different momentum curves in this manner — such as linear or nonlinear momentum growth, etc. At the end of the slideshow we give each subject all the slides and ask them to prioritize them in order of most important to least important — we ask them to tell us what they think the most important slides in the slideshow were. This effectively tests the various distributions we ran in the experiment to see which ones had the strongest cognitive effect on the subjects. Two weeks or a month later we repeat this rating test to see which distributions have the strongest long-term effect as well. By doing this experiment many times with many distributions we can experimentally determine which memetic distributions have the strongest cognitive impact. The next step would be to test whether the distributions we discover are applicable across sensory modalities — for example, do the distributions we found for vision also work for the auditory system. My hypothesis is that they do hold across modalities. If this is the case then we have discovered a key underying meta-pattern in the human perceptual system — the pattern by which humans recognize what to tune their attention to.

There is another interesting and related question to the above experiments: Do certain distributions retain attention better than others? The human perceptual system attenuates to signals very quickly — we tune out anything regular or predictable and focus on identifying novelty. But what is “novelty?” Any new meme that occurs is novel at first, but whether or not it remains novel or gets tuned out is another question. Which meme distributions do NOT get tuned out as quickly, or ever? Is there an optimal way to vary the distribution of a meme such that it continues to remain novel? In thinking about this, are there any meta-patterns to the memes that have gotten your attention in the past? For example, is there something about the way that particular technology trends or celebrities have moved through the media that made them appear to be hotter and more important to you? Having high momentum at a given time is part of this, but it may in fact be the change in momentum over time — the “meme impulse” — that really makes the difference. For example in my own experience I notice that trends that exhibit exponential growth in momentum quickly get my attention — but as soon as the growth becomes predictable I lose interest. So it seems that the trends that retain my interest the best are the ones that have more variable graphs — graphs that are neither random nor regular. Is there an ideal balance between randomness and order? What patterns have this balance — can we quantify this and define it more concretely?

A better understanding of the cognitive effects of various higher order distributions of memes in various human sensory modalities could be particularly useful for advertisers, marketers, and user-interface designers. An advertiser or marketer could use this knowledge to design campaigns that get the most attention and that are not “tuned out” by people as quickly. A user-interface designer could use this information to design interfaces for manging changing information in which the signal-to-noise ratio is optimized so that users can quickly focus on just the most important changing information — for example the information display of a stock-trading terminal, executive information system, military situation room, or fighter jet cockpit user-interface could perhaps be improved using these principles.

Concluding Remarks

Given that memes are now among the most powerful “hidden” forces shaping our individual minds, our relationships, organizations and our world, wouldn’t it be great if we could really measure them and analyze them empirically?

That is what I hope the basic techniques provided above will help to catalyze. By making this hidden layer visible we can gain a much better understanding of our world. Let me know if you end up using these techniques for anything interesting (and hopefully you will make your ideas open-source too so everyone can benefit).

What these basic techniques provide is a way to measure the movement of ideas in time and space. For example, we can track the trajectories of ideas in our workspaces, our teams, enterprises, cities, nations or interest-communities. We can also track them across geography or any other set of dimensions.

Because we can compute basic physical properties of memes we can start to apply Newtonian physics to analyze them. Perhaps by doing so we can really develop a “Physics of Memetics” with which we may begin to predict the outcomes of interactions among memes, the future trajectories of memes, and the influence changes to memes have on events in the so-called “real world” and vice-versa. With this in hand we could potentially teach systems to learn to detect memetic patterns of interest to us — for example the early “fingerprints” in the media that indicate the outcome of a proposed act of legislation or a vote, or a stock price, or a political change. We could also use it to detect emerging cultural trends, and to measure and compare the dynamics of brands or competing technologies in various markets in order to predict winners.

By putting this information into the public domain I hope to see these techniques in use as widely as possible. They will provide dramatic benefits in managing large volumes of information, improving knowledge worker and team productivity, and in discovering and measuring trends in communities.

Ultimately, I would like to see this embodied in a “grand cultural project” — a real-time map of the memetic dynamics taking place around the globe. This map would be filterable in order to show relative memetic dynamics in different places, communities, etc., and to show how various memes are spreading and interacting over time around the world. The data would be open and accessible via an open API so that all services that manage information could provide information to it and query it for stats when needed.

Minding the Planet: From Semantic Web to Global Mind

Nova — Sun, 27 Jun 2004 07:02:35 +0000

Draft 1.1 for Review (integrates some fixes from readers)
Nova Spivack (www.mindingtheplanet.net)

INTRODUCTION

This article presents some thoughts about the future of intelligence on Earth. In particular, I discuss the similarities between the Internet and the brain, and how I believe the emerging Semantic Web will make this similarity even greater.

DISTRIBUTED INTELLIGENCE

The Semantic Web enables the formal communication of a higher level of language — metalanguage. Metalanguage is language about language — language that encodes knowledge about how to interpret and use information. Metalanguages – particularly semantic metalanguages for encoding relationships between information and systems of concepts – enable a new layer of communication and processing. The combination of computing networks with semantic metalanguages represents a major leap in the history of communication and intelligence.

The invention of written language long ago changed the economics of communication by making it possible for information to be represented and shared independently of human minds. This made it less costly to develop and spread ideas widely across populations in space and time. Similarly, the emergence of software based on semantic metalanguages will dramatically change the economics not only of information distribution, but of intelligence — the act of processing and using information.

Semantic metalanguages provide a way to formally express, distribute and share the knowledge necessary to interpret and use information, independently of the human mind. In other words, they make it possible not just to write down and share information, but also to encode and share the background necessary for intelligently making use of that information. Prior to the invention of such a means to share this background knowledge about information, although information could be written and shared, the recipients of such information had to be intelligent and appropriately knowledgeable in advance in order to understand it. Semantic metalanguages remove this restriction by making it possible to distill the knowledge necessary to understand information into a form that can be shared just as easily as the information itself.

The recipients of information – whether humans or software – no longer have to know in advance (or attempt to deduce) how to interpret and use the information; this knowledge is explicitly coded in the metalanguage about the information. This is important for artificial intelligence because it means that expertise for specific domains does not have to be hard-coded into programs anymore — instead programs simply need to know how to interpret the metalanguage. By adding semantic metalanguage statements to information data becomes “smarter,” and programs can therefore become “thinner.” Once programs can speak this metalanguage they can easily import and use knowledge about any particular domain, if and when needed, so long as that knowledge is expressed in the metalanguage.

In other words, whereas basic written languages simply make raw information portable, semantic metalanguages make knowledge (conceptual systems) and even intelligence (procedures for processing knowledge) about information portable. They make it possible for knowledge and intelligence to be formally expressed, stored digitally, and shared independently of any particular minds or programs. This radically changes the economics of communicating knowledge and of accessing and training intelligence. It makes it possible for intelligence to be more quickly, easily and broadly distributed across time, space and populations of not only humans but also of software programs.

The emergence of standards for sharing semantic metalanguage statements that encode the meaning of information will catalyze a new era of distributed knowledge and intelligence on the Internet. This will effectively “make the Internet smarter.” Not just monolithic expert systems and complex neural networks, but even simple desktop programs and online software agents will begin to have access to a vast decentralized reserve of knowledge and intelligence.

The externalization, standardization and sharing of knowledge and intelligence in this manner, will make it possible for communities of humans and software agents to collaborate on cognition, not just on information. As this happens and becomes increasingly linked into our daily lives and tools, the "network effect" will deliver increasing returns. While today most of the intelligence on Earth still resides within human brains, In the near future, perhaps even within our lifetimes, the vast majority of intelligence will exist outside of human brains on the Semantic Web.

THE INTERNET IS A BRAIN AND THE WEB IS ITS MIND

Anyone familiar with the architecture and dynamics of the human nervous system cannot help but notice the striking similarity between the brain and the Internet. But is this similarity more than a coincidence – is the Internet really a brain in its own right – the brain of our planet? And is its collective behavior intelligent – does it constitute a global mind? How might this collective form of intelligence compare to that of an individual human mind, or a group of human minds?

I believe that the Internet (the hardware) is already evolving into a distributed global brain, and its ongoing activity (the software, humans and data) represents the cognitive process of an increasingly intelligent global mind. This global mind is not centrally organized or controlled, rather it is a bottom-up, emergent, self-organizing phenomenon formed from flows of trillions of information-processing events comprised of billions of independent information processors.

As with other types of emergent computing systems, for example John Conway’s familiar cellular automaton “The Game of Life,” on the Internet large scale homeostatic systems and seemingly intentional or guided information processes naturally emerge and interact within it. The emergence of sophisticated information systems does not require top-down design or control, it can happen in an evolutionary bottom-up manner as well.

Like a human brain, the Internet is a vast distributed computing network comprised of billions of interacting parallel processors. These processors include individual human beings as well as software programs, and systems of them such as organizations, which can all be referred to as "agents" in this system. Just as the computational power of the human brain as a whole is vastly greater than that of any of the individual neurons or systems within it, the computational power of the Internet is vastly beyond any of the individual agents it contains. Just as the human brain is not merely the sum of its parts, the Internet is more than the sum of its parts – like other types of distributed emergent computing systems, it benefits from the network effect. The power of the system grows exponentially as agents and connections between them are added.

The human brain is enabled by an infrastructure comprised of networks of organic neurons, dendrites, synapses and protocols for processing chemical and electrical messages. The Internet is enabled by an infrastructure of synthetic computers, communications networks, interfaces, and protocols for processing digital information structures. The Internet also interfaces with organic components however – the human beings who are connected to it. In that sense the Internet is not merely an inorganic system – it could not function without help from humans, for the moment at least. The Internet may not be organized in exactly the same form as the human brain, but it is at least safe to say it is an extension of it.

The brain provides a memory system for storing, locating and recalling information. The Internet also provides shared address spaces and protocols for using them. This enables agents to participate in collaborative cognition in a completely decentralized manner. It also provides a standardized shared environment in which information may be stored, addressed and retrieved by any agent of the system. This shared information space functions as the collective memory of the global mind.

Just as no individual neuron in the human brain could be said to have the same form or degree of intelligence as the brain as-a-whole – we individual humans cannot possibly comprehend the distributed intelligence that is evolving on the Internet. But we are part of it nonetheless, whether we know it or not. The global mind is emerging all around us, and via us, is our creation but it is already becoming independent of us – truly it represents the evolution of a new form of meta-level intelligence that has never before existed on our planet.

Although we created it, the Internet is already far beyond our control or comprehension – it surrounds us and penetrates our world – it is inside our buildings, our tools, our vehicles, and it connects us together and modulates our interactions. As this process continues and the human body and biology begins to be networked into this system we will literally become part of this network – it will become an extension of our nervous systems and eventually, via brain-computer interfaces, it will be an extension of our senses and our minds. Eventually the distinction between humans and machines, and the individual and the collective, will gradually start to dissolve, along with the distinction between human and artificial forms of intelligence.

MEMES ARE EVOLVING MINDS OF THEIR OWN

The evolution of our planetary intelligence has been taking place for billions of years — it is a natural process, just like the evolution of human intelligence was long ago. The Semantic Web is merely the next step in this process whereby communicable ideas (memes), having already evolved technologies to externalize themselves outside the human mind (i.e. books, recording, software, the Web, etc.) are starting to evolve the ability to propagate intelligently and interact without human intervention. In other words, although today memes are for the most part completely immobile and static unless perceived within a human brain, with the advent of the Semantic Web the cognitive processes for running memes will begin to spread outside the human brain, enabling memes to "run" without depending on humans.

This emerging planet-wide collective mind, of which we will be but parts, will evolve higher level meta-processes and structures that will vastly exceed our comprehension. Indeed this is already starting to happen — even today the self-organizing, chaotically emergent collective intelligence and information flows of the Internet exceed the power and understanding of any computer or brain on the planet. This new meta-level intelligence will be as far beyond human intelligence as the intelligence of the human brain is beyond that of its individual neurons.

THE INFRASTRUCTURE OF DISTRIBUTED INTELLIGENCE

The development of the global mind depends on the evolution of distributed systems that function as the global equivalent of consciousness, memory, learning, perception, introspection, planning, creativity, and behavior.

Distributed intelligence requires the decentralization of information and computation. The World Wide Web is a key catalyst for this evolutionary leap. Before the Web there was no universally agreed-upon standard for publishing and accessing simple information – instead there were myriad incompatible, non-standardized competing proprietary formats. The lack of a common language made it difficult for applications to interoperate or understand one another’s data without explicit integration.

The significance of the Web is that its underlying metalanguage standards – HTML and HTTP – enable more widespread, interoperable and decentralized content production and access. Making it possible for agents anywhere in the system to publish and make use of information by any other agents in the system is an essential ingredient of a distributed intelligence. The Web is literally a World Wide File System – it is the memory function of the global mind.

If the Web enables the World Wide File System, the emergence of XML enables The World Wide Database. XML enables agents in the system to define, store, retrieve, interact with, and interpret arbitrary data structures with arbitrary precision. Using XML any conceivable syntax and data schema can be defined and shared. XML adds more structure to the information in the memory of the global mind, enabling more sophisticated content and processes to be stored and accessed by agents in the system.

The recently emerging Semantic Web adds yet another layer of sophistication beyond XML. It enables agents in the system to begin to understand and reason about the meaning of information within the system. The Semantic Web enables software to work not merely with data but with concepts. Concepts are information structures that are connected to formal systems of ideas – in other words they are meaningful information. The Semantic Web provides standards for transforming ordinary information structures into concepts that can be understood by software programs. Using metalanguages for defining semantics such as RDF and OWL, the Semantic Web makes it possible to connect data elements to concepts in formally defined systems of knowledge called ontologies. By doing this software programs are able to then reason intelligently about the information.

By connecting information to ontologies, programs can begin to process information more intelligently. For example, the content of a medical journal could be linked to a medical ontology that defines medical concepts and their interrelations. Using this ontology it would then be possible to do semantic searches of the journal that are far more intelligent than the primitive keyword searches that are currently used in most search systems today. A semantic search for “information about the vascular system" would return articles and data records that refer to the heart, even though the word "heart" was not explicitly searched for. Furthermore, a semantic search for "organs connected to the heart" could make logical inferences across chains of concepts in the underlying medical ontology in order to return articles about the lungs, the liver, the kidneys, the brain, etc., even though none of those organs were explicitly named in the original query.

Smarter searches are just one of the many benefits of the Semantic Web. Beyond such basic applications, the Semantic Web makes it possible for software to automatically learn, reason, make suggestions, and manage tasks and processes more intelligently. What’s more, by providing a standardized language for describing systems of concepts and chains of reasoning, the Semantic Web makes it possible for programs to seamlessly share concepts and collaborate on reasoning tasks – in other words, it makes it possible not only for smarter computation within a given program, but it also enables smarter computation to take place between programs, making it possible for widespread distributed artificial intelligence to emerge on the Internet.

THE EVOLUTION OF METALANGUAGE

The Semantic Web is based on a higher level of language — metalanguage — language about language. Metalanguage is a form of communication that enables parties to rigorously express and share information about the meaning of information. In fact, metalanguage has existed since the dawn of humanity. For example, in the case of spoken language, humans communicate metalanguage by using tone, gesture, inflection, volume, and facial expressions. These cues convey vital information about the meaning of what we are communicating, making it possible for those we communicate with to more easily understand us. In written language very simple forms of metalanguage have also been in use for quite some time, such as for example, the formatting of text, the use of footnotes and diagrams. The way text is organized on a page, and the particular typefaces and styles used also constitute metalanguage expressions about the meaning of the text.

The Semantic Web provides metalanguage specifications and technologies that vastly increase the bandwidth and sophistication of metalanguage communication for all forms of digital media. For example, using metalanguages such as XML, RDF and OWL, the Semantic Web makes it possible to encode arbitrarily detailed knowledge about the structure, meaning, state, connections, reliability, sentiment, and policies of arbitrary chunks of information. In other words, a document can be encoded with metalanguage that adds layers of additional knowledge about the information it contains. These layers of information augment the text — they may provide definitions, links to other resources, information about the organization of information within the document, logical relations among concepts in the document, details about the history and license terms of the document, annotations from other readers, and even rules for interpreting, reasoning about, or using the document. What is important here is that this metalanguage is expressed in a manner that machines can understand.

In effect, semantic metalanguage gives computers access to layers of knowledge that previously could only exist or be utilized within the human brain. By making this metalanguage explicit and by standardizing it, it becomes possible not only to communicate it effectively between humans, but also between humans and programs, and even between programs and other programs.

The evolution from simple typography to SGML and HTML to XML and finally to the Semantic Web (RDF and OWL) can be viewed as a process of decoupling the interpretation of data from the agents that produce and consume the data. In other words not only the data itself, but also its interpretation, can now be stored outside of the agents of communication. HTML makes it possible for any program to correctly render data. XML makes it possible for any program to correctly parse and navigate the structure of data – for example to find a particular data element such as a field within a document. RDF and OWL make it possible for any program to understand what a particular data element means, and to reason about it.

If we look back to the dawn of humanity there was a time when humans were only able to communicate nonverbal or primitive verbal information. As richer forms of communication evolved sophisticated spoken languages and oral traditions emerged enabling the communication of more complex ideas. But spoken language had a major limitation – the distribution and access to information was dependent on being physically proximate enough to interact with particular individuals.

With the development of written languages however, it became possible to break through this limitation. Writing systems made it possible for ideas to be represented, stored and communicated independently of any particular individual, with less error, across greater distances in space and time than ever before. For the first time it was possible to learn something from someone else without them having to be present – anyone who could read the language and had sufficient background could interpret written characters into concepts. Next, with the advent of printing the economics of distributing and accessing written ideas reached a critical threshold of efficiency, enabling widely distributed communication and intellectual discourse.

Centuries later another critical threshold was crossed with the invention of long-distance communications networks such as teletypes, telephones, radio and television. These technologies made communication faster, richer, broader, and more ubiquitous and accessible than ever before. As recorded and recordable media emerged even these rich media experiences could be experienced asynchronously anywhere and at any time.

Next, the emergence of computers and computer networks made it possible for communications and information processes to be increasingly automated. At this point we begin to see something new – while previously only information could be represented outside of the human brain now even primitive forms of intelligence (information processing) could be represented and conducted outside of the human brain. The Internet and the World Wide Web are the logical extension of this process – they make it possible to distribute and access information, and to connect information and processors together, more widely than ever before – but they still rely on humans for the most part.

Without humans the Internet and Web of today would be nothing but a collection of relatively static information and dumb computer systems. But XML and the Semantic Web will change that by providing metalanguages that make it possible for humanlike intelligence to being to evolve and function outside of human brains. With advent of metalanguages humans are no longer necessary to create or interpret information. These technologies will enable the Web to actively and intelligently process information without human participation.

Metalanguages such as HTML, XML, RDF and OWL enable knowledge about information to be formally encoded into the information itself. As increasing levels of knowledge about data is encoded into the data, the data becomes more independent of humans - it can be used by any agent anywhere.

HOW THE GLOBAL MIND THINKS

Semantic Web programs will share and process information intelligently, with or without the help of humans, by reading and writing metadata about data in a standardized way such that other programs can then reuse it. Programs will be able to leverage the knowledge that other programs create about the data they work with – even though these programs may not be directly integrated or even know about one another. In effect the Web becomes a gigantic shared knowledgebase that every program can read and write to.

Just as colonies of social insects such as ants and bees are able to perform intelligent collective behaviors without centralized control, the millions, or even billions, of humans and programs roaming independently through the Semantic Web, selectively reading, writing, annotating, linking, rating, and aggregating information, will perform collective intelligent behaviors without necessarily coordinating with one another or even knowing it. In other words the individual agents in such behaviors will participate in collective cognitive processes that transcend the comprehension of any individual.

Here’s how it might work: Imagine that a particular news article about a potential corporate merger exists on the Web. Intelligent agents – whether humans or software programs - are then able to read this article and mark it up with semantic metadata in their particular areas of expertise. One agent specializes in identifying company names – whenever it sees the name of a company in an article it tags it with a link to the ontology definition of a corporation, as well as with metadata that links it to the Web site and other data records corresponding to the particular corporation it represents. Another agent specializes in recognizing people: whenever it sees the name of a person it tags it with a link to the ontology class for “person" and also with metadata that connects it to the home page for that person, articles about that person, friends and colleagues of that person, organizations that the person is affiliated with etc. Other agents that visit, or receive, the article could then tag it with their particular knowledge – some add metadata about links, others tag events, others add metadata about places, others add metadata about products and brands, others add metadata about technical terms and jargon, etc.

We might even imagine that some of these agents are capable of generating new articles and data structures about the original article and linking them together – for example, one agent might generate a synopsis, another might translate it into another language, another might measure the opinions in the article, still another might generate a report based on the conclusions in the article. Because all of this knowledge is expressed using open semantic metadata standards, any program that later encounters any of it can make use of it in its own work, without having to be expressly programmed to do so.

This is already starting to happen in fact – For example, in the blogging community and communities of practice, which in an entirely bottom-up emergent manner, are naturally aggregating, annotating, linking, organizing and prioritizing information. Although there is no central guidance within such knowledge communities, their collective self-organizing behavior results in global information processes that appear to be intelligent. If one were to view the information dynamics of the Web from space – perhaps with a special sensor that could detect and measure these patterns as they emerged – would it not appear similar to the a functional brain imaging scan?

The Internet (the OS layer), the Web (the data layer), XML (the data schema and syntax layer), and the Semantic Web (the knowledge and reasoning layer) combine to provide the foundation for an increasingly intelligent distributed world-wide mind. They enable all the agents of the global mind to seamlessly share not just raw information, but even high-level concepts, knowledge and intelligent cognitive processes, in a manner that is open and independent of any individual system.

In particular the Semantic Web makes it possible to represent concepts such that they can be unambiguously interpreted and understood by any agent of the system. However, the success of this process will hinge on the development and adoption of open-standards-based, open-source ontologies, and mappings between them. This is already starting to take place, for example, FOAF, a simple ontology for describing social relationships, and SUMO, a standardized ontology of foundational concepts, among many others. I believe much of the initial development of these much-needed open-source ontologies will spring from the Weblog and RSS communities, where there is an increasing willingness (and need) on the part of participants to mark up and filter content with metadata.

CAN THE GLOBAL MIND PASS THE TURING TEST?

If the Internet is becoming a global mind, is there a way to test whether or not it is actually intelligent? Of course that first requires that one define intelligence – a notoriously fuzzy term! For the purposes of this article, we might define intelligence as “humanlike information processing.” One way to test for “humanlike intelligence” is to use the Turing Test – in which a human subject attempts to determine which of two “black boxes” is controlled by a human and which is a computer in a question-and-answer game.

An interesting modern-day spin on the classic Turing Test, might test large distributed online communities comprised of people and software programs, to see if such systems could be judged to be intelligent. It seems like a good bet that such systems – if hidden in a black box would be able to emulate “humanlike intelligence.”

I once tested this hypothesis in my own company many years ago. A difficult math problem was posed to me and to the best mathematician in our team. Whoever could answer it correctly the fastest would be judged as the best mathematician. I have never been much of a mathematician, but I still won this contest. My strategy was simply to farm out the problem to a number of the best mathematical brains I knew, integrate the answers, and package it up as a reply to the question.

My network of math-brains vastly outperformed the ability of my own brain or the brain of the math expert I was competing with. Not knowing how I solved the problem, those in the company would only be able to assume that I was a better mathematician. In point of fact however it was not a fair contest. "I" was not merely an individual but a vast collective super-brain comprised of several networked experts. The other guy was hopelessly outgunned.

This is an example of the power of distributed intelligence – the world of the future that is evolving on the Web right now. As the global brain continues to develop we will see individual humans, and even individual organizations, being dramatically outpaced by collective intelligences. One compelling example of how this is happening can be seen in the rise of open-source software development communities which are able to develop better code, faster, at less cost, and with broader adoption than has ever been possible by single entities.

READING THE GLOBAL MIND

If the Web is becoming a virtual mind of the planet is it possible to data-mine the Web in order to empirically measure, map, understand and even predict collective cognition? Can we empirically measure the Web in order to chart the past and present thinking of individuals, groups and communities, nations, or even of humanity-as-a-whole? By dong this can we learn to detect and track thoughts ("memes") as they emerge, spread, interact, develop and evolve in real-time? If we are able to empirically detect memes and develop a science of meme dynamics would this enable us to not only better understand the past and the present, but even to predict the future in a new way?

One approach to reading the global mind is to measure distributed cognitive trends by mining search engines results for the frequency of search terms over time, such as Google’s Zeitgeist reports. More recent approaches such as Daypop attempt to detect "word bursts" on the Web and "news bursts" among news articles.

Many academic and government research projects have explored the potential to data-mine news articles and other information sources in order to predict political events. For example, this project found that it is possible to predict conflicts such as wars as early as 6 to 8 weeks before they occur by data-mining news articles. There have also been projects to predict signs of political change, such as coups and election results, by data-mining political news.

Another interesting project describes a technique for statistically analyzing clusters of concepts that occur on the Web in order to attempt to find hot archetypes in the collective consciousness of humanity. The particular application of this system that the authors focus on is predicting terrorist events. Their system identifies hot themes, but requires a high degree of subjective interpretation in order to come up with predictions. While interesting, I am not sure the system can be used to reliably predict the future, although it certainly can help to understand the present. In any case, this project is significant in that it attempts to detect collective thoughts or archetypal patterns that transcend any individual mind or community. It’s definitely worth reading for those interesting in next-generation data-mining.

Another project that takes a completely different approach is the Global Consciousness Project which mines statistical deviations from randomness across a network of random number generators around the world and then correlates these deviations with global events — nobody knows why this works but the statistical data speaks for itself (this project may in fact point to yet another interesting connection between consciousness and quantum physics, similar to the famous double-slit experiment but on a global scale, but nobody really knows — all we know so far is that the data is sound.) This project might be described as an EEG for the planet. While it cannot provide insight into particular thoughts taking place in the global mind, it does provide a window into the activation and dynamics of the global mind.

In my own thinking on the subject, I have focused more on detecting and analyzing the higher-order distribution of memes in space and time. Memes are concepts that move across the global mind – they the building blocks of its collective thoughts. A meme might be as simple as a brand or an icon, or as sophisticated as a joke, a fact, tradition, fad, belief system, or a paradigm. I have spent some time speculating about a possible physics of ideas that might be able to empirically detect, measure and predict the dynamics and interactions among memes on the Web. My approach attempts to measure properties of memes in space and time in order to forecast their trajectories. For example, using this approach, one might be able to measure the geographic footprint, mass and velocity of a meme over time. With such data it then becomes possible to begin to measure the spread of ideas much like one might analyze the behavior of systems of particles, or the behavior of products and stocks within marketplaces.

The examples above represent just a small sample of the many research projects and technologies in this space. I would be very interested to hear of others of note.

ENTERPRISE MINDS

As the global mind develops it will initially be focused around making information more useable. But that will be just the beginning. Already a new generation of tools that will bring the power of distributed intelligence to the desktop and the enterprise are being developed in labs such as HP, Cycorp and Network Inference.

In addition to these projects, my company,Radar Networks has developed a complete platform in Java for developing and deploying Semantic Web applications.

The power of distributed intelligence made possible by the Semantic Web will dramatically evolve corporations – at least those that are not made extinct by it. In particular, it will enable workgroups and corporations to begin to distill and store not only their information but also their intelligence. As individuals and teams work, intelligent agents will learn from them. These agents will then be able to assist them in working more productively — they will help them search, organize and file information, track relevant news, better-leverage existing knowledge and resources, manage projects and tasks, share and access knowledge, and communicate and collaborate more productively with teammates, customers and partners. Similarly, smart agents will learn from corporations as-a-whole, and from their business interactions with employees, customers, suppliers and partners in order to dynamically streamline business processes and adapt to market changes intelligently.

By making organizational knowledge, learning and intelligence increasingly independent of the particular minds or programs within a given organization, all parts of these organizations as well as the whole organizations themselves will become more intelligent. As knowledge and intelligence about organizations become increasingly portable and reusable, organizations will evolve their own "group minds" and "enterprise minds." These distributed forms of intelligence will constitute a new level of structure, a new layer of organization. Such meta-level processes will help managers make smarter decisions by enabling them to better access the combined past and present knowledge and capabilities within their organizations and business relationships. They will also help organizations to notice opportunities or problems, and response to them more effectively.

Today there are many organizations that have realized that their primary product is knowledge. Tomorrow organizations will begin to realize that it is not just knowledge, but also intelligence, that is the key to their competitive advantage. Intelligence is the ability to utilize knowledge effectively.

Merely creating vast collections of knowledge that are inaccessible or simply not leveraged is of no benefit to anyone. What matters is that the knowledge is intelligently connected to business processes such that it measurably improves performance. What is necessary for this to happen is not merely the implementation of knowledge management systems, but rather the implementation of intelligent systems — a new way of creating and utilizing knowledge at all levels of the organization.

Knowledge must be intelligently integrated into every business activity, event, relationship, resource and tool. Furthermore the integration must be bidirectional — every business activity should be able to get knowledge from the enterprise and add knowledge back to it. By enabling this, with the right infrastructure and tools, organizations can literally begin to learn and improve based on their own collective experience. By providing all of the parts of an organization with access to the collective knowledge and intelligence of the system, the whole system can become more collectively intelligent.

At a higher-level, in order to enable more focused, goal-directed collective behaviors it is necessary to create control structures and adaptive feedback loops between the "parts" and the "wholes" within an organizations. What this means is that there needs to be a connection between the knowledge and intelligence taking place within each part, and the new meta-level knowledge and intelligence taking place at the level of the whole (such as a team or enterprise). The question then arises as to how to bring about such a connection? What connects the parts to the whole — what makes a collection of parts function as a whole, yet enables them to still maintain their individuality and independence? What enables the whole to function as one entity, despite being formed of myriad independent parts?

Traditional control structures such as top-down management hierarchies err on the side of the whole — they attempt to rigidly organize and control the parts of the organization in order to force them to conform into a cohesive whole. On the other extreme, more recent attempts to eliminate hierarchy and enable flatter, more "networked" organizations err on the side of the parts — they eliminate the hierarchical control structures altogether leaving nothing but chaotically interacting decentralized parts. Fortunately there is another alternative — there is a way to connect the parts and the whole without sacrificing either. The key is enabling richer self-knowledge.

In order to bring about synchrony between levels of a distributed organization there must be three essential ingredients: (1) The state of each part must be represented, (2) the state of the whole system (the combined system of all the parts) must be represented, (3) the parts must all have real-time access to all of these representations. By meeting these three requirements feedback becomes possible in several directions — between each part and every other part, and between each part and the system as-a-whole. These representations and feedback loops provide a vital function to distributed intelligences — they enable them to enact a simple form of self-awareness. Self-awareness is a vital ingredient of higher forms of intelligence. The richer a systems’ self-representation, the smarter and more effective it can be.

In order to accomplish self-awareness in a highly distributed organization, each part of the organization needs to have access to a self-representation of itself as well as of the whole system. Each part needs to be able to understand themselves and the system they are part of. By providing the parts of a distributed organization with access to information about both their own state and the state of the whole, the parts are empowered to adapt to the whole. By enabling this, the whole is also more able to adapt to the parts, because there is bidirectional feedback taking place between these levels. Rather than placing control structures at only the level of the whole, or at only the level of the parts, instead they are distributed across both levels.

The only way for this to be practical, economically feasible, or even technically possible, is by using emerging Semantic Web metalanguages. These metalanguages provide a common standard for sharing knowledge and intelligence at every level and across every part of an organization. Knowledge and intelligence are thus able to move freely across and between them and organizational learning takes place on the individual level, the group or sub-systems level, and the level of the whole system. And this learning is expressed, stored and shared in a single common metalanguage that is equally accessible to all. This is quite different from the case of present-day organizations in which there are different languages and formats for knowledge at different levels of the organization. For example, in most present-day organizations human knowledge and expertise is still locked inside individual human minds and totally dependent on them, group knowledge is stored on PCs and workgroup servers, and enterprise knowledge is stored in enterprise systems. Each of these systems speaks a different language and most are not directly integrated.

Numerous inefficiencies result from this. Why should it be so difficult to move a concept across an organization? Why does it require that data be translated and ported from one person to another, from one program to another? The reason it is so difficult today is that the interpretation of the data is not stored separately from the brains and programs that manipulate it — in other words, metalanguages are not being used. As a result, the intelligence of the organization is not portable — it is locked into silos such as people’s heads and particular applications that are explicitly programmed with particular skills and knowledge.

Those organizations that understand this are already starting to make use of metalanguages and Semantic Web technologies. Those that are first to begin exploring and deploying these "enterprise minds" will have a valuable head start that may provide them with crucial advantages in the marketplace. This is not unlike the advantages that Homo sapiens had over earlier primates. Larger and more advanced brains resulted in an increased capacity for language, communication and reasoning that ultimately enabled them to outperform less intelligent hominids. This same principle holds for organizations.

CONCLUDING THOUGHTS

The ideas in this essay are not unique to me – they are memes that are spreading on their own through the global mind. Many others such as the people involved with the Principia Cybernetica Project or my friend Howard Bloom have thought far more extensively than I have about these subjects. In writing this article I am merely providing a service to the global mind – that of aggregating, annotating and communicating these memes onward in a process that I cannot begin to comprehend. All I know is that the global mind is thinking about its own evolution and realizing that it is intelligent – and that I am just an infinitesimal part of that process. Yet, like you who are reading this, I somehow sense that what is taking place is incredibly important and will change our world and our species profoundly.

The Memecodes Project

Nova — Fri, 07 May 2004 16:34:06 +0000

The Memecodes Project starts with randomly generated Web pages and evolves them to get more search results from Google. Brilliant idea. The only problem is that Google indexing isn’t frequent enough, making this process take a bit of time. Still, I think this is a very interesting new approach that could even be a useful utility for sites in general someday.