If 2010 was the year of the Real-Time Web, then 2011 is going to be the year that it evolves into the Scheduled Web.

The Real-Time Web happens in the now: it is spontaneous, overwhelming, and disorganized. Things just happen unpredictably and nobody really knows what to expect or what will happen when.

The Real-Time Web is something of a misnomer, however, because usually it’s not real-time at all –  it’s after-the-fact. Most people find out about things that happened on the Real-Time Web after they happen, or, if they are lucky, when they happen. There is no way to know what is going to happen before it happens; there is no way to prepare or ensure that you will be online when something happens on the Real-Time Web. It’s entirely hit-or-miss.

If we are going to truly realize the Real-Time Web vision, then “time” needs to be the primary focus. So far, the Real-Time Web has mainly just been about simultaneity and speed – for example how quickly people on Twitter can respond to an event in the real world such as the Haiti Earthquake or the Oscars.

This obsession with the present is a sign of the times, but it is also a form of collective myopia — the Real-Time Web really doesn’t include the past or the future – it exists in a kind of perpetual now. To put the “time” into Real-Time, we need to  provide a way to see the past, present and the future Real-Time Web at once.  For example, we need a way to search and browse the past, present, and the future of a stream – what happened, what is happening, and what is scheduled to happen in the future. And this is where what I am calling The Scheduled Web comes in. It’s the next step for the Real-Time Web.

Defining the Scheduled Web

With the Scheduled Web things will start to make sense again. There will be a return of some semblance of order thanks to schedule metadata that enables people (and software) to find out about upcoming things on the Web that matter to them, before they happen, and to find out about past things that matter, after they happen.

The Scheduled Web is a Web that has a schedule, or many schedules, which exist in some commonly accessible, open format. These schedules should be searchable, linkable, shareable, interactive, collaborative, and discoverable. And they should be able to apply to anything — not just video, but any kind of content or activity online.

Why is this needed? Well consider this example. Imagine if there was no TV Guide on digital television. How would you navigate the constantly changing programming of more than 1000 digital TV channels without an interactive program guide (IPG)? It would be extremely difficult to find shows in a timely manner. According to clickstream data from television set-top boxes, about 10% of all time spent watching TV is spent in the IPG environment. And that is not even counting additional time-spent in on-demand guidance interfaces on DVRs. The point here is that guidance is key when you have lots of streams of content happening over time.

Now extend this same problem to the Web where there are literally millions of things happening every minute. These streams of content are not just limited to video. There are myriad types of real-time streams, everything from sales, auctions, and chats, to product launches, games, and audio, to streams of RSS feeds, Web pages appearing on Web sites, photos appearing on photo sites, software releases, announcements, etc.

Without some kind of guidance it is simply impossible to navigate the firehose of live online content streams on the Web efficiently. This firehose is too much to cope with in the present moment, let alone the past, or the future. This is what the Scheduled Web will solve.

By giving people a way to see into the past, present and future of the Real-Time Web, the Scheduled Web will enable the REAL Real-Time Web to be truly actualized. People will be able to know and plan in advance to actually be online when live events they care about take place.

Instead of missing that cool live Web concert or that auction for your favorite brand of shoes, simply because you didn’t know about it beforehand, you will be able to discover it in advance, RSVP, and get reminded before it starts — so you can be there and participate in the experience, right as it happens.

We are just beginning to see the emergence of the Scheduled Web. Two new examples of startups that are at work in the space are Clicker and Live Matrix.

• Clicker, a site that mainly provides on-demand video clips of past TV episodes, this week launched a schedule for live video streams on the Web.

• Live Matrix (my new startup), is soon to launch a schedule for all types of online events, not just video streams.

Some people have compared Live Matrix to Clicker, however this is not a wholly accurate comparison. We have very different, although  intersecting, goals.

While Clicker is an interesting play to compete with TV Guide and companies like Hulu, Live Matrix is creating a broader index of all the events taking place across the Scheduled Web, not just video/TV content events.

The insight behind Live Matrix is that there is much more to the Scheduled Web than video and TV content. The Web is not just about TV or video – it is about many different kinds of content.

Applying a TV metaphor to the Web is like trying to apply a print metaphor to tablet computing. While print has many positive qualities, tablet devices should not be limited just to text should they? Likewise, while the TV metaphor has advantages, it doesn’t make sense to limit the experience of time or scheduled content on the Web just to video.

With this in mind, while Live Matrix includes scheduled live video streams, we view video and TV type content as just one of many different types of scheduled Web content that matter.

For example, Live Matrix also includes online shopping events like sales and auctions, which comprise an enormous segment of the Scheduled Web. As an illustration eBay alone lists around 10 million scheduled auctions and sales each day! Live Matrix also includes scheduling metadata for many other kinds of content — online games, online chats, online audio, and more.

Live Matrix is building something quite a bit broader than current narrow conceptions of the Real-Time Web, or the narrow metaphor of TV on the Web. We are creating a way to navigate and search the full time dimension of the Web, we are building the schedule of the Web.

This will become a valuable, even essential, layer of metadata that just about every application, service and Internet surfer will make use of every day. Because after all, life happens in time and so does the Web. By adding metadata about time to the Web, Live Matrix will help make the Web – and particularly the Real-Time Web – easier to navigate.

Online vs. Offline Events

One of the key rules of Live Matrix is that, to be included in our schedule, an event must be consumable on-line. This means that it must be possible to access and participate in an event on an Internet-connected device.

Live Matrix is not a schedule of offline events or events that cannot be consumed or participated in using Internet-connected devices.

We made this rule because we believe that in the near-future almost everything interesting will, in fact, be consumable online, even if it has an offline component to it. We want to focus attention on those events which can be consumed on Internet-connected devices, so that if you have a connected device you can know that everything in Live Matrix can be accessed directly on your device. You don’t have to get in your car and drive to some physical venue, you don’t have to leave the Internet and go to some other device and network (like a TV and cable network).

Note the shift in emphasis here: We believe that the center of an increasing number of events is going to be online, and the offline world is going to increasingly become more peripheral.

For example, if a retail sale generates more revenues from online purchases than physical in-store purchases, the center of the sale is really on-line and the physical store becomes peripheral. Similarly, if a live concert has 30,000 audience members in a physical stadium but 10,000,000 people attending it online, the bulk of the concert is in fact online. This is already starting to happen.

For example, the recent Youtube concert featuring U2 had 10 million live streams – that’s up to 10 million live people in the audience at one time, making it possibly the largest online concert in history; it’s certainly a lot more people than any physical stadium could accommodate. Similarly, online venues like Second Life and World of Warcraft can accommodate thousands of players interacting in the same virtual spaces – not only do these spaces not even have a physical analogue (they exist only in virtual space), but there are no physical spaces that could accommodate such large games. These are examples of how online events may start to eclipse offline events.

I’m not saying this trend is good or bad; I’m simply stating a fact of our changing participatory culture. The world is going increasingly online and with this shift the center of our lives is going increasingly online, as well. It is this insight that gave my co-founder, Sanjay Reddy, and I, the inspiration to start Live Matrix, and to begin building what we hope will be the backbone of the Scheduled Web.

The emerging realtime Web is not only going to speed up the Web and our lives, it is going to bring about a kind of awakening of our collective Global Brain. It’s going to change how many things happen on online, but it’s also going to change how we see and understand what the Web is doing. By speeding up the Web, it will cause processes that used to take weeks or months to unfold online, to happen in days or even minutes. And this will bring these processes to the human-scale — to the scale of our human “now” — making it possible for us to be aware of larger collective processes than before. We have until now been watching the Web in slow motion. As it speeds up, we will begin to see and understand what’s taking place on the Web in a whole new way.

This process of of quickening is part of a larger trend which I and others call “Nowism.” You can read more of my thoughts about Nowism here. Nowism is an orientation that is gaining momentum and will help to shape this decade, and in particular, how the Web unfolds. It is the idea that the present-timeframe (“the now”) is getting more important, shorter and also more information-rich. As this happens our civilization is becoming more focused on the now, and less focused on past or the future. Simply keeping up with the present is becoming an all-consuming challenge: Both a threat and an opportunity.

The realtime Web –  what I call “The Stream”  (see “Welcome to the Stream”) — is changing the unit of now. It’s making it shorter. The now is the span of time which we have to be aware of to be effective our work and lives, and it is getting shorter. On a personal level the now is getting shorter and denser — more information and change is packed into shorter spans of time; a single minute on Twitter is overflowing with potentially relevant messages and links. In business as well, the now is getting shorter and denser — it used to be about the size of a fiscal quarter, then it became a month, then a week, then a day, and now it is probably about half a day in span. Soon it will be just a few hours.

To keep up with what is going on we have to check in with the world in at least half-day chunks. Important news breaks about once or twice a day. Trends on Twitter take about a day to develop too. So basically, you can afford to just check  the news and the real-time Web once or twice a day and still get by. But that’s going to change.  As the now gets shorter, we’ll have to check in more frequently to keep abreast of change. As the Stream picks up speed in the middle of this decade, to remain competitive will require near-constant monitoring — we will have to always be connected to, and watching, the real-time Web and our personal streams. Being offline at all will risk missing out on big important trends, threats and opportunities that emerge and develop within minutes or hours. But nobody is capable of tracking the Stream all 24/7 — we must at least take breaks to eat and sleep. And this is a problem.

Big Changes to the Web Coming Soon…

With Nowism comes a faster Web, and this will lead to big changes in how we do various activities on the Web:

• We will spend less time searching. Nowism pushes us to find better alternatives to search, or to eliminate search entirely, because people don’t have time to search anymore. We need tools that do the searching for us and that help with decision support so we don’t have to spend so much of our scarce time doing that. See my article on “Eliminating the Need for Search — Help Engines” for more about that.
• Monitoring (not searching) the real-time stream becomes more important. We need to stay constantly vigilant about what’s happening, what’s trending. We need to be alerted of the important stuff (to us), and we need a way to filter out what’s not important to us. Probably a filter based on influence of people and tweets, and/or time dynamics of memes will be necessary. Monitoring the real-time stream effectively is different from searching it. I see more value in real-time monitoring than realtime search — I haven’t seen any monitoring tools for Twitter that are smart enough to give me just the content I want yet. There’s a real business opportunity there.
• The return of agents. Intelligent agents are going to come back. To monitor the realtime Web effectively each of us will need online intelligent agents that can help us — because we don’t have time, and even if we did, there’s just too much information to sift through.
• Influence becomes more important than relevance. Advertisers and marketers will look for the most influential parties (individuals or groups) on Twitter and other social media to connect with and work through. But to do this there has to be an effective way to measure influence. One service that’s providing a solution for this (which I’ve angel invested in and advise) is Klout.com – they measure influence per person per topic. I think that’s a good start.
• Filtering content by influence. We also will need a way to find the most influential content. Influential content could be the content most RT’d or most RT’d by most influential people. It would be much less noisy to be able to see only the more influential tweets of people I follow. If a tweet gets RT’d a lot, or is RT’d by really influential people, then I want to see it. If not, then only if it’s really important (based on some rule). This will be the only way to cope with the information overload of the real-time Web and keep up with it effectively. I don’t know of anyone providing a service for this yet. It’s a business opportunity.
• Nowness as a measure of value of content. We will need a new form of ranking of results by “nowness” – how timely they are now. So for example, in real-time search engines we shouldn’t rank results merely by how recent they are, but also by how timely, influential, and “hot” they are now. See my article from years ago on “A Physics of Ideas” for more about that. Real-time search companies should think of themselves as real-time monitoring companies — that’s what they are really going to be used for in the end. Only the real-time search ventures that think of themselves this way are going to survive the conceptual paradigm shift that the realtime Web is bringing about. In a realtime context, search is actually too late — once something has happened in the past it really is not that important anymore –what matters is current awareness: discovering the trends NOW. To do that one has to analyze the present, and the very recent past, much more than searching the longer term past. The focus has to be on real-time or near-real-time analytics, statistical analysis, topic and trend detection, prediction, filtering and alerting. Not search.
• New ways to understand and navigate the now. We will need a way to visualize and navigate the now. I’m helping to incubate a stealth startup venture, Live Matrix, that is working on that. It hasn’t launched yet. It’s cool stuff. More on that in the future when they launch.
• New tools for browsing the Stream. New tools will emerge for making the realtime Web more compelling and smarter. I’m working on incubating some new stealth startups in this area as well. They’re very early-stage so can’t say more about them yet.
• The merger of semantics with the realtime Web. We need to make the realtime Web semantic — as well as the rest of the Web — in order to make it easier for software to make sense of it for us. This is the best approach to increasing the signal-to-noise ratio of content we have to look at whether searching or monitoring stuff. The Semantic Web standars of the W3C are key to this. I’ve written a long manifesto on this in “Minding The Planet: The Meaning and Future of the Semantic Web” if you’re really interested in that topic.

Faster Leads to Smarter

As the realtime web unfolds and speeds up, I think it will also have a big impact on what some people call “The Global Brain.” The Global Brain has always existed, but in recent times it has been experiencing a series of major upgrades — particularly around how connected, affordable, accessible and fast it is. First we got phone and faxes, then the Internet, the PC and the Web, and now the real-time Web and the Semantic Web. All of these recent changes are making the Global Brain faster, more richly interconnected. And this makes it smarter. For more about my thoughts on the Global Brain, see these two talks:

• My detailed History and Future of the Global Brain given at the Singularity Summit.
• A talk on the emerging Global Brain and human-machine cybernetic superorganism, with specific focus on what it means for media companies, from the GRID Conference.

What’s most interesting to me is that as the rate of communication and messaging on the Web approaches near-real time, we may see a kind of phase change take place – a much smarter Global Brain will sort of begin to appear out of the chaos. In other words, the speed of collective thinking is as important to the complexity or sophistication of collective thinking, in making the Global Brain significantly more intelligent. In other words, I’m proposing that there is a sort of critical speed of collective thinking, before which the Global Brain seems like just a crowd of actors chaotically flocking around memes, and after which the Global Brain makes big leaps — instead of seeming like a chaotic crowd, it starts to look more like an organized group around certain activitities — it is able to respond to change faster, and optimize and even do things collectively more productively than a random crowd could.

This is kind of like film, or animation. When you watch a movie or animation you are really watching a rapid series of frames. This gives the illusion of there being cohesive, continuous characters, things and worlds in the movie — but really they aren’t there at all, it’s just an illusion — our brains put these scenes together and start to recognize and follow higher order patterns. A certain shape appears to maintain itself and move around relative to other shapes, and we name it with a certain label — but there isn’t really something there, let alone something moving or interacting — there are just frames flicking by rapidly . It turns out that after a critical frame rate (around 20 to 60 frames per second) the human brain stops seeing individual frames and starts seeing a continuous movie. When you start flipping pages fast enough it appears to be a coherent animation and then we start seeing things “moving within the sequence” of frames. In the same way, as the unit of time of (aka the speed) of the real-time Web increases, its behavior will start to seem more continuous and smarter — we won’t see separate chunks of time or messages, we’ll see intelligent continuous collective thinking and adaptation processes.

In other words, as the Web gets faster, we’ll start to see processes emerge within it that appear to be cohesive intelligent collective entities in their own right. There won’t really be any actual entities there that we can isolate, but when we watch the patterns on the Web it will appear as if such entities are there. This is basically what is happening at every level of scale — even in the real world. There really isn’t anything there that we can find — everything is divisible down to the quantum level and probably beyond — but over time our brains seem to recognize and label patterns as discrete “things.” This is what will happen across the Web as well. For example, a certain meme (such as a fad or a movement) may become a “thing” in it’s own right, a kind of entity that seemingly takes on a life of its own and seems to be doing something. Similarly certain groups or social networks or activities they engage in may seem to be intelligent entities in their own rights.

This is an illusion in that there really are no entities there, they are just collections of parts that themselves can be broken down into more parts, and no final entities can be found. However, nonethless, they will seem like intelligent entities when not analyzed in detail. In addition, the behavior of these chaotic systems may resist reduction — they may not even be understandable and their behavior may not be predictable through a purely reductionist approach — it may be that they react to their own internal state and their environments virtually in real-time, making it difficult to take a top-down or bottom-up view of what they are doing. In a realtime world, change happens in every direction.

As the Web gets faster, the patterns that are taking place across it will start to become more animated. Big processes that used to take months or years to happen will happen in minutes or hours. As this comes about we will begin to see larger patterns than before, and they will start to make more sense to us — they will emerge out of the mists of time so to speak, and become visible to us on our human timescale — the timescale of our human-level “now. As a result, we will become more aware of higher order dynamics taking place on the real-time Web, and we will begin to participate in and adapt to those dynamics, making those dynamics in turn even smarter. (For more on my thoughts about how the Global Brain gets smarter, see:  “How to Build the Global Mind.”)

See Part II: “Will The Web Become Conscious?” if you want to dig further into the thorny philosophical and scientific issues that this brings up…