Joshua Schachter, the creator of Delicious, has launched his newest creation, Jig.

At first glance the site seems a bit like Twitter, but it has a different focus. Instead of posting about what you are doing, you post about what you need. Then other people reply with suggestions, ideas, answers, help, or presumably commercial products and services that can meet your need.

This is not a new idea. It’s been done before, at least in print, quite successfully, in the form of “the want ads.” Want ads are classified ads, where instead of offering something, you ask for something. They are basically inverse classified ads. Like a reverse auction is an inverse auction.

But although it’s not groundbreakingly new, it’s beautifully executed and quite simple and elegant. It’s elegant enough in fact that it might catch on. And if it does, it could be quite useful.

The site has some similarities to Quora, but it’s broader. It’s not just about questions and answers – it’s about getting help with any kind of need.

Looking through the initial needs being posted by early users there are requests for restaurants suggestions, a guy asking what gift he should buy for his minimalist girlfriend, a request to understand how UFO propulsion works, requests to hire people, and even a request for affordable health insurance.

There also seems to be quite a bit of spam, or at least unhelpful questions and comments, including some harmless but irrelevant banter. Jig will need to provide for a way to rank needs, comments, and authors so that noise is filtered out. This is a problem that Schachter should be able to solve in his sleep, so I’m not worried about that being a barrier to adoption. It will be resolved soon, I’m betting.

There’s a lot of potential here, if people actively start helping to share their tips and advice for getting needs met. One challenge will be to make it easy for people to find needs they can help with. A categorization system, based on hashtags perhaps, would help to find needs that match your offers or areas of expertise.

All the product level issues are pretty easy to solve. This is not rocket science. But a harder problem to solve is, how is Jig going to make money? Who is going to have to pay for what? There’s always a catch somewhere. At least if the goal is to build a revenue business.

Will users eventually be charged to post certain kinds of needs? Or is the idea to charge companies, for example, as they are asked to do when posting job ads in Craigslist? Or will there be some kind of reverse auction or group buying angle to this – when enough people have the same need they can pool together and negotiate for a group deal?

Time will tell. But since it’s Joshua Schachter, Jig is bound to get a lot of attention. Check it out for yourself and see if it meets your needs.

By the way, if you’re reading this, tell our reporters at The Daily Dot (@dailydot) what you think of Jig, and whether it’s helped you in any interesting ways. We’re curious to hear your perspective.

Today I’m pleased to announce that, The Daily Dot, our newest “venture production,” has launched into public beta.

The Daily Dot is the first of its kind – it’s the Web’s newspaper — the first community newspaper about the Web. We cover the Web like a town paper covers its community. Here’s a video overview of the site.

This venture began with the insight that each of us is spending an increasing amount of our lives online, in various online communities, yet we have very little insight into what’s going in this new landscape. These communities are literally places, and some of them are quite large. This is beautifully illustrated in this “map” of the Web as a geography.

I believe that it’s time for the Web community to have it’s own newspaper. The launch of the Daily Dot — the web community’s first actual newspaper of record — is a turning point, a coming-of-age, for the Web as a medium, as a place, and as a community.

Our editorial focus is different than other publications that cover the Web. Instead of covering the Web as an industry, a technology or a phenomenon, we cover it as a community. We tell the stories of the people, culture, content, events and issues that are making waves in communities around the Web. And to find and report on these stories, we have embedded reporters in those communities: Facebook, YouTube, Reddit, Twitter, Tumblr, with more communities coming soon.

Just like our physical cities and towns, our online communities are constantly moving and developing, and they are full of interesting people doing newsworthy and important things. The Daily Dot’s mission is to cover these communities just like physical community newspapers cover cities and towns.

Where a town newspaper covers the latest high school sports game, the town meeting, the local crime report, we cover the story behind the hottest viral video sweeping the planet, the latest social movement in Facebook, and important issues (like cybercrime or online bullying) that are happening in our online neighborhoods.

When a major event happens in the physical world – like the revolutions in Arab world, for example — we don’t cover the events themselves, we cover their online footprint — what’s happening online that relates to the story.

The Daily Dot will also cover what’s happening around the Web in time: just like physical community newspapers have calendar sections – The Daily Dot has an online events section, provided in partnership with Live Matrix, one of our other venture productions, that aggregates the schedule of the Web. These two companies are highly synergistic and form the beginnings of our online media network.

While those of us in the Web industry have our fingers slightly more on the pulse of the Web, the vast majority of people who use the Web do not read industry blogs and have little or no visibility into what’s going on in the online world or where it’s headed. Other than a few articles a week published by mainstream media, they are not being informed.

It’s time for that to change. The Daily Dot will be publishing dozens of articles each day about what’s happening online. We’re writing for the mainstream, not for elites or geeks. The Daily Dot is for the people who use the Web — who live in it — not just the people who are building it.

Our content is designed to be entertaining, interesting, informative — and sometimes edgy and controversial – kind of like People Magazine meets USA Today, with a little bit of TMZ thrown in.

If you want to know what’s happening online, or you’re looking to find the hottest emerging entertainment, personalities, viral videos, issues, etc — and the stories behind them — The Daily Dot is your newspaper.

But The Daily Dot is not just a newspaper, it’s also a very interesting business venture. It’s a chance to build what could become one of the largest circulation newspapers in the world someday – a global newspaper about the one community that we all share in common, no matter where we actually live.

I also want to congratulate and thank the amazing editorial and development team at the Daily Dot, who made this possible. And most importantly, I want to acknowledge Nicholas White (Daily Dot CEO), Owen Thomas (Daily Dot founding editor), and Josh Jones-Dilworth (marketing guru), my co-founders in this venture.

Nick and Owen are leading business and editorial, and running the operations, and Josh and myself are on the board, advising to help in our respective areas of expertise. Nick and Owen deserve all the credit here — they have done the heavy lifting to bring this vision to market, and I’m very proud to be working with them.

Please join me us helping to spread the word about The Daily Dot — it’s your newspaper — and we need your help to make it great (and we look forward to your feedback and participation in the comments).

This is going to be a fun ride and I can’t wait to see how it evolves.

The social media landscape is changing quickly, but this change won’t be immediate, or for that matter, efficient. And that’s going to be a big problem for all of us.

I believe that Twitter, Facebook, Google+ and LinkedIn are fundamentally different, and thus, should not be in competition. However, I’m not sure the companies themselves see it this way. It’s likely they will continue dedicating resources to competition instead of differentiation.

And while the social media gods fight it out in the clouds above us, what will happen down here on Earth? What about all of us, the little people — the users?

We’re entering a new era of social network chaos, and this, in turn, is going to create new needs and opportunities for startups.

The Sharepocalypse

Welcome to the “Sharepocalypse,” a new era of social network insanity.

READ THE REST OF THE ARTICLE HERE

TechCrunch kindly ran my most recent article today — the full version is available here.

Here is an excerpt:

I’ve been puzzling over Twitter’s recent tactical moves around their API, Ubermedia and Tweetdeck, for a few months now, and it just doesn’t add up. In fact I think Twitter’s current strategy may take them in a direction where they end up missing out on their biggest potential win.

If Twitter continues to go down the media company path, without incorporating their API into the plan, that could not only force a large part of their ecosystem to go elsewhere, but it could deprive them of a much larger potential infrastructure revenue opportunity, and could even end up costing them the company.

After all, Silicon Valley is littered with the  burned out wreckage of once-great media companies that failed create and keep third-party app ecosystems: AOL, Friendster, MySpace, Yahoo – to name a few. It’s very hard to maintain leadership as an online media company without an ecosystem of outside apps increasing reach, innovation, and stickiness.

In light of this, I’ve been exploring an alternate path for Twitter that leverages their API in a much bigger way, and this path appears to be a better strategy. According to my own experimental revenue  projections for Twitter, this alternative path is not only a good tactical move, but it’s a good business move because it increases Twitter’s reach, number of active users, and revenues massively.

….. Read the rest here.

I’m pleased to announce that my newest venture production is beginning to unstealth. It’s called The Daily Dot and it promises to be “the hometown newspaper of the Web ” — the community newspaper for Web.

The story of The Daily Dot began several years ago when I was thinking about where the Web was headed. At the time I was thinking a lot about the future of emerging online communities such as Digg, Facebook, the early days of Twitter, and even my own Twine.com — as well as about the growth of fully immersive games and virtual worlds like World of Warcraft and Second Life. I realized that these communities really were virtual places, and some of them literally even contained the equivalent of towns, leagues, guilds, nations.

But when I looked at how the media was covering the Web at the time, I saw a huge gap. Coverage broke out into two areas: stories aimed at an industry audience (TechCrunch, ReadWriteWeb, GigaOm, Venturebeat, Techmeme) and coverage aimed at an early-adopter tech audience (Wired, Engadget, Slashdot, Boing Boing). But collectively, these audiences made up only a small slice of the overall Web audience pie.

Even more notable was that nobody was covering online communities like places — the way that newspapers cover nations, cities and towns. There were no local reporters, embedded reporters,  no stringers or correspondents in various online communities. In short, traditional media was covering the Web like a technology, not like a place.

Where was the coverage for the majority of the audience? The mainstream consumers who spent the better part of every day in this new place we call the Web? Where was the coverage of what was happening in the communities on the Web? The stories about the people on the Web? The stories for the people who used the Web?

Curiously, when I dug into this, I found that the mainstream was receiving scattered attention, in the form of only a few human interest or business articles per week in the major national media outlets (New York Times, Wall Street Journal, CBS, NBC, ABC). Mostly these articles were either curiosities or they were about big financial deals around hot companies. They too failed to address the Web as a place.

While the Internet industry audience was being deluged with thousands of geeky articles and blog posts every day, the mainstream audience was for the most part being ignored by the media. My mainstream audience member friends confirmed this – they had no clue at all about what was really going on online – even around topics they cared about like brands, celebrities, music, major privacy issues that would affect them, the birth and death of major online services, new social trends and memes, new legislation, cybercrime. The small amount of this news they were aware of, reached them weeks after it was fresh, when the major outlets finally covered it.

I realized that here was an opportunity – in fact a need – for a newspaper that covered news about the Web for the people who use the Web — mainstream people. A newspaper by and for the people of the Web (in other words, all of us). A newspaper that would cover the Web like a place and as a community. In further discussions about this concept, my wife, Kimberly Rubin, came up with the perfect name, “The Daily Dot” and I went about buying the domain name.

The idea gestated and grew. I couldn’t stop thinking about it. Finally, I decided this was a good enough idea to actively produce it in my new “venture production studio” — And so with that in mind I began looking for the right CEO and co-founders to produce the venture around.

As fate would have it, I had been introduced to Nicholas White through my longtime friend and PR guru, Josh Jones-Dilworth. Nick and I had been circling for a while. He had this “young Richard Branson” vibe — which everyone comments on after they meet him. I knew he was going to be someone important but I wasn’t sure exactly in what way.  Then it hit me.

Nick grew up in a newspaper family, working in the print newspaper biz. For over a century his family has been running community newspapers; Today they own 22 newspapers and radio stations. Like me and Josh, Nick had been thinking about the same problem — how to evolve community news reporting for the new millennium, but from the perspective of the newspaper business.

Nick was thinking about how to save the newspaper business — thoughts which he has elaborated on this week in  a new article about how he hopes to save the newspaper business by leaving it. As we spoke about the Daily Dot and his own ideas, I realized Nick had both the pedigree and the passion to build what I had envisioned. Nick was the perfect CEO for the Daily Dot. And so we invited him to co-found the venture.

With Nick’s experience at the helm, we are already making great progress. An example of this is last week’s announcement that the widely-followed editor, Owen Thomas, has left his position as executive editor at Venturebeat (a terrific publication that I read every day), to join Nick and the team as founding editor of the Daily Dot. And around Nick and Owen we are already growing a team of really awesome editors, writers, designers, coders, marketers and investors. It’s really starting to take shape, rapidly.

Owen in particular brings a strong editorial background, and is already helping to focus our strategy. We were all impressed by Owen’s incredible network of connections to the movers and shakers of the Web, as well as to the users of the Web — and also with his knowledge of all things media, pop culture, gossip, fashion, design, entertainment and more. He really understands what people use the Web for, and he’s got a great nose for news. In short he’s got exactly the right mix to head up the Daily Dot’s editorial strategy.

As Owen explains it, The Daily Dot, is going to cover the Web in a new way: It’s about people. We’re going to cover the Web not just as a technology or an industry, but as a community — actually a community of communities — spread across a virtual landscape of online places. Some of these communities, like Facebook, are even larger than physical nations, and contain communities within them that are larger than many cities. Others, like World of Warcraft, are complex parallel worlds complete with warring factions and their own economies.

And there are many other vibrant communities on the Web: Youtube, Etsy, Second Life, 4Chan, the Word Press blogging community, Tumblr, Reddit, and literally millions of micro communities around vertical interests. These communities have people in them — yes actual people, not just technologies and venture capitalists! And these people have stories, stories we want to know about. And so do the people who participate in them. But who is telling those stories?

Imagine a nation or city without its own daily newspaper — how would people know what’s going on, what would hold it together, would it even feel like one nation or city at all? A newspaper is a critical enabling catalyst that transforms a crowd into a community. It gives people news, but also a sense of place, a sense of belonging, a sense of community. It tells the story of the place, it holds the record of the place. On a deeper level, a newspaper provides a mirror of the whole back to the parts, enabling an essential feedback loop. In short, newspapers are the lifeblood of communities.

The Web today is like that nation or city without a newspaper. It’s missing something essential – the one key catalyst that will transform it from a crowd into an actual community. By providing the Web with its own newspaper, The Daily Dot will make the Web feel more human, more connected, and more cohesive. And this is really important.

The Daily Dot aims to be the community newspaper for the Web as a whole, as well as for each of the communities within it. And by doing this, we may just end up play a key role in the life of the Web. But that’s the just the beginning: We may also become the first truly global newspaper; the newspaper with the largest daily readership on the planet. After all, what newspaper today has 6 billion daily readers?

There is no geographic print newspaper audience that large. But The Daily Dot is not limited by geography; it has a real chance at achieving a truly global readership by covering the one community that everyone on this planet has in common: The Web. It’s the first newspaper that everyone may actually read every day.

The Daily Dot is still young – in fact we haven’t really even launched it yet. And as we launch it’s going to be a work in progress: We’ll be starting with a series of experiments, a newsletter, and some explorations of new approaches for involving the community in making its own news, and then we’ll be launching a major new site — currently in private beta.

Meanwhile, we’re hiring writers and editors, so if you share our passion for this mission (and you’re awesome) definitely apply. We look forward to hearing your stories.

I’m writing this post since many of my friends and colleagues have gotten wind of some news and asked me what I’m up to. This is just the first in a series of articles I’ll be writing on this topic.

In a nutshell, I’ve been working behind the scenes for the last year to co-found and angel invest in a number of exciting new ventures. Several of these ventures will be launching soon, and so it’s time to begin telling the story of what they do, and the big idea behind them: a new approach to building startups that borrows from how Hollywood produces movies.

And in keeping with this, I’ve moved with my wife Kimberly Rubin (a TV producer with 11 movies to her credit), from San Francisco to Los Angeles, where Hollywood production studios began. I believe LA is a great place to build this concept out.

I call this new model of venture incubation the “production studio model” and in this approach I work as a producer of ventures, not merely a founder or angel investor.

As well as being a better fit for the needs of early stage startups than the typical angel investor or VC approach, the production studio model has enabled me to start a number of really excellent ventures, for less cost, in less time, than I ever thought possible.

Production Portfolio

But before I go into more detail about the model, here is the current portfolio of companies that I am actively producing. With the exception of Live Matrix and Klout (which were started earlier), all of these companies were started in the last year and will be launching soon:

• Live Matrix — The schedule of the Web. Live Matrix is the only guide to what’s happening, when, online – across all media types (video, audio, chat, gaming, shopping, and more). I co-founded and seed-funded this venture with CEO, Sanjay Reddy, and I continue to incubate it and actively participate, in the same way as I have since we began, by serving on the board and helping on product strategy, marketing and technology. Live Matrix is launched and busy making deals and launching new features. More news coming soon.
• Klout — Klout is the standard for measuring influence. I discovered the company when I was a judge at the SXSW Accelerator in 2009. I was really impressed with the founders and soon became the company’s first outside investor. Since then I have served actively as an advisor to the company. They recently raised a terrific venture round with Kleiner Perkins and are off to the races.
• Bottlenose — You’ll be hearing a lot about this venture soon. It’s a new personalization technology designed for the realtime Web and social media. I co-founded this company with Dominiek ter Heide. As well as seed-funding the company, I’m taking an increasingly active role in helping to build this company.
• The Daily Dot — The Daily Dot is a new online newspaper about the Web, for consumers. Most of the coverage of the Web today is targeted at the tech industry, a tiny fraction of the audience, but the Daily Dot will cover the Web for the majority of the audience: consumers who spend much of their day, every day, online. The Daily Dot hasn’t launched yet but it’s going to be an exciting company. I co-founded it with newspaper-industry CEO, Nicholas White and co-founder Josh Jones-Dilworth. More news will be coming out soon!
• StreamGlider — StreamGlider is a new visual real-time dashboard for tracking interests across various types of devices, starting with the iPad. It’s got a gorgeous user-interface and some novel features that are especially suited to keeping up with streams of rich media. I co-founded this venture with two leading technologists in the information filtering and semantics space, Bill McDaniel and John Breslin. This will be launching soon as well. More news will be available at launch.
• “Project Nikola” This new venture has a breakthrough new energy technology and is totally in stealth. This isn’t even its real name; even that is a secret, for now. What I can say currently is that it really works, it’s mind-blowingly cool and just may disrupt the entire power grid someday. But there’s still a lot of R&D to do before we release it.
• The Earth Dashboard. This is a not-for-profit initiative (yes, I sometimes help produce game-changing nonprofits too) that is working to create an interactive live dashboard about the state of the planet that brings together, and visualizes, all the key global indicators, in one place for the first time. This project is led by Medard Gabel, who worked with Buckminster Fuller, and the creative director is Mia Hanak and her accomplished museum exhibit design team.The Dashboard will be available online as well as in major physical public locations around the globe.

Several of these companies have a common thread – and a common passion for me — they are focused on helping people filter the Web in potentially disruptive ways. Some are using “Big Data” analytics, data mining and extraction, natural language processing, machine learning and semantics, to understand the Web. These are areas that I am deeply familiar with from my many years working around information filtering, AI and search. The Nikola project is an exception — it is outside of the Internet space but springs from a multi-decade interest I’ve had in radical alternative energy technologies.

A History of Incubation

Since 1994, I’ve been involved in starting companies as an entrepreneur, and since 2000 I’ve also been an angel investor. Through incubating numerous ventures (my own and those I’ve angel invested in), I’ve gained some experience into the art of incubating startups.

But one of the best experiences I had was starting one of the more successful incubators, nVention, at SRI, which I conceived of and co-founded with Norman Winarsky (now head of ventures at SRI) in 1999. nVention is now global and has launched more than 40 ventures.

Unlike many incubators, nVention acts in a very hands-on way. First of all, most of nVention’s work focuses around creating ventures to commercialize intellectual property that was originated at SRI. Secondly, nVention bring teams of internal and external experts together to help incubate its companies from concept stage through commercialization. In effect, nVention acts like a production studio, and the people who work there function like producers. It’s a model I’m emulating, albeit in a more grassroots and distributed way.

The Venture Production Model

As a producer, I work actively to develop new original intellectual property, or to source it from great innovators, and then I angel invest and/or bring funding to the deal, shape products and strategies, build teams, invent and develop products, and actively grow companies and take them to market. In many cases, the ventures I’m producing are originated by me, but I also have several ventures in my portfolio that were originated by others, or in partnership with others.

The key to this approach is that I usually get involved at or even before concept stage — even before there is a real team — and I actively work to shape it into a venture, from concept through commercialization.

To accomplish this, across more than one venture at a time, I partner with excellent people to co-produce these companies. In some cases I act as the startup CEO, in other cases my partners do, or we find and partner with the right person to be CEO. Often I find myself partnering with entrepreneurs and helping to coach them to be CEO’s. But in all of these cases, we focus on producing ventures together, as a team. And we’re all in it for the long-term. Because I’ve found excellent partners to work with, including excellent CEO’s where needed, I’m personally able to focus more intensively on helping each venture. This has worked very well so far.

The production approach to venture creation is quite different from the “fire and forget” or “spray and pay” (or “pay and pray”) model that many VC’s and angel investors are engaging in. Instead of spreading lots of fairly hands-off bets across dozens of companies, in the production model I really focus and get deeply hands-on with a pipeline of projects of various stages.

This the opposite of the index fund or hedge fund approach that some funds are taking in the Valley. And I think it is a much better fit for the needs of early stage companies.

The production approach is also different from what many incubators and start-up accelerators are doing these days. Incubators and accelerators play an important role in the startup ecosystem, but the key difference is that in the Hollywood-inspired production model I’m testing, I often start earlier in the process – before there is a concept, company, product, CEO or even a team. Many incubators and accelerators start later in the process.

Here’s how it works. My associates and I source candidate ideas both from our own stream of inventions, as well as from people in our network, and from other innovators we find or who find us. Next we “option” the best ideas with joint R&D agreements and/or with initial prototype funding to test them out. We then filter these prototypes and choose the best ones to produce.

We then form companies, build teams, develop business plans, branding and strategy, and bring additional funding together to develop the commercial offerings, launch, market, and grow them into full ventures. It’s a very hands-on process – and just like movie production, and we usually have a number of projects going on at each stage in our pipeline at once.

The process is similar to producing a film. With a film, first you have to create or find the story, then hire writers and a director, recruit talent, and build a production team, get the financing and early distribution deals in place, shoot the film, do post-production, get broader distribution, market the film and release it.  In the early stages of companies, we all wear many hats, and as they grow, we specialize. In my own case, I assume different roles and levels of involvement according to the needs of the ventures as they grow.

The New Role of the Producer in Tech

A key to this process is really in understanding the new role of a producer in the venture world. It’s not exactly the same as the role of an angel or VC, or an EIR, or even a typical “superhero CEO.” It’s a new role that connects them all together.

Another key is having a model that is designed to attract excellent producers and talent to team up with. To do that, I’m working with a structure that gives my partners a better opportunity than they can find anywhere else. Whether it is an externally or internally originated venture, the model I’m working with is, by far, the most entrepreneur-friendly deal in the entire industry.

In most of my ventures today I take a minority, or at most, an equal partner position, with my cofounders – even in the ventures I originated and funded. In some of the ventures I have angel funded, I have continued to maintain the original equity split with my co-founders, even as the amount of funding I have contributed has increased over time. Where other angels and VC’s take the approach that money is everything, I take the exact opposite view. Talent is everything. Talent is rare, and it’s the lifeblood of ventures.  I don’t believe it is healthy for any company to have the investors take control away from founders — too often that results in disaster. My model is all about cultivating and facilitating the founders.

Why do I do this? Because I believe that you get the best out of people when they really feel they own their venture, and when they feel respected and valued for their contributions. Part of this is because as an entrepreneur myself I have experienced life on the other side. I know intimately what it is to be an entrepreneur, and I know how some VC’s, and even some angels, take advantage of entrepreneurs and founders, suck the life out of companies, and destroy businesses by over-controlling them, and I vowed NOT to be like that.

The terms I offer are the same terms I always wanted for myself. No bullshit and no games. We all succeed or fail together. It’s a true partnership. I’m not betting the odds across dozens of companies and expecting only 1 or 2 to survive; I’m NOT doing shotgun investing like most angels out there, I’m making extremely careful, extremely deeply involved long-term commitments to build companies side-by-side with my partners. And I don’t just talk about this, I do it – my model reflects this.

At the same time, I also spend money in a different way than angels or VC’s. For example, in many of my projects I’ll start on spec with a developer and an idea. No money is initially invested. They work on the idea and prove they have the goods. Then if that works, there is a small grant to “option” it and we develop it further, much like a production studio options a story. I then work with the technical or product teams to see what they can deliver with this initial grant – usually a prototype. This is a test.

If the test goes well and progress is good, we make a production deal in which money, and time, are invested in stages, by myself and others in my network, as needed, rather than all at once. We’re not talking about huge dollar investments early-on, it’s frugal and careful, but it’s enough, and it’s extremely value-added. Later in the process, when the time is right, more money can be brought to work.

Building out the Production Studio

Another way I help my portfolio companies is by bringing pre-negotiated deals with handpicked best-of-breed vendors for many of the services they need. I bring the top law firms and patent teams, the best PR and marketing teams in the business, and accounting and HR services, from partners I know and trust. This saves ventures valuable time and money, and protects them from making early mistakes.

Once the ventures are aimed at a clear target and have something to show, we go together to other angels and venture funds in my network to raise the roll-out money. There’s no reason to give up equity until we need to. The result is that entrepreneurs who work with me end up owning bigger stakes of their ventures than they would if they worked with traditional angels and VC’s.

On the investment side, I’ve been meeting with interesting angel investors – some are pros and some are new to this – and we’re teaming up to jointly fund these ventures together. By working with a production team such as mine, angel investors can put their money to work with less hands-on effort on their parts – because we’re doing the production work for them. That doesn’t mean they aren’t involved – we consult with them as much as they want and we actively solicit their feedback and ideas at every step — but it means they can have confidence that an experienced team is doing the groundwork day-to-day. By working with producers like myself and my team, angels and funds can spread their bets without being spread too thin. If you’re an angel investor, or even a VC, and you’re curious about working with our network, drop me a line.

On my own side, of course, I have to be very picky about what ventures I get involved with, so that I and my team are not spread too thin as well. For that reason, we only allow for about 4 ventures in any stage (from concept stage, through R&D, to beta, and commercialization) of our pipeline at once. But this is a model that I think can scale as we add more producers and team members to the mix. Scaling this is an area that I am actively thinking about right now. If you’re an exceptional venture producer and you would like to be involved, get in touch. This is not about anyone being a superhero, it’s about creating an awesome and highly collaborative team, supported by an incredible network.

There’s so much more I could say about all this, and in time, I’ll write more about it.You will be hearing a lot more about this model, as well as all of the ventures we’re producing in coming months, as well as some new ventures not listed above that are in the pipeline and will become visible in the future.

In the meantime, I’ll be posting here and on my Twitter feed, about my thoughts and what we’re learning as this goes forward. I welcome your thoughts too — this is one of the reasons I’ve written this. So please don’t hesitate to ask questions, offer advice, or observations. Stay tuned, it’s going to be an adventure!

– Nova

Tonight I am pleased to announce that my next Big Idea has launched. It’s called Live Matrix and I invite you to come check it out.

Live Matrix is the schedule of the Web — We help you to find out “What’s When on the Web” — the hottest live online events happening on the Web: concerts, interviews, live chat sessions, game tournaments, sales, popular Webshows, tech conferences, live streaming sports coverage, and much more.

It’s like TV Guide was for TV, but it’s not for TV, it’s for the Web. There are all kinds of things happening online — and while Live Matrix includes a lot of live streaming video events, there is much more than just video in our guide. Live Matrix includes any types of scheduled online events — but we don’t include offline events — to be in Live Matrix an event must enable people to participate online.

The site combines elements of a guide, a search engine, and a DVR, to help you discover events and then get reminded to attend them, or catch them later if you missed them.

The insight that led to Live Matrix was that the time-dimension of the Web is perhaps the last big greenfield opportunity on the Web. It’s an entire dimension of the Web that nobody has made a search engine for, and nobody is providing any guidance for. Nobody owns it yet — it’s a whole new frontier of the Web.

There are millions of scheduled events taking place online every day. Some of these events are very cool, some are very relevant — but there is no easy way to find out about them. To find out what’s happening when on TV for example, we have TV Guide, but there is no equivalent for finding out what’s happening when on the Web.

In my own case I kept finding out about cool online events that I would have participated in — concerts, conference streams, webinars, online debates and interviews, and sales –  if only I had known they were happening. I think many Internet users have experienced this.

Google, Yahoo and Bing all focus on what I call the “space dimension” of the Web — they help you find what’s where — where is the best page about topic x? — But they don’t help you find out what’s when — what’s happening now, what’s coming next. They only help you find out what’s already finished and done with. How do you find out what’s happening now? How do you know what’s upcoming?

It was an “aha moment” when this all became clear — there is a new opportunity to be the Google or Yahoo for the time dimension of the Web. Or at least to be the equivalent of a TV Guide for the Web.

Furthermore, All trends point to this being a big opportunity. The continued growth of the realtime Web (Twitter, etc.) and the emerging Live Web (video and audio streaming) has been discussed extensively in the media; most recently comScore reported nearly a 650% increase in time spent viewing live video online.

So with this opportunity clearly in mind I set about looking for a co-founder who would be the right person to team up with, someone who would be the CEO.

That person was Sanjay Reddy. Soon after I met Sanjay it was clear to me that he was the exact right guy to partner with: his background in media and technology were what impressed me (for example, he was head of corp dev, strategy and M&A at Gemstar-TV Guide, where he led the $2.3 billion dollar sale of the company to Macrovision, and he had also worked at other Silicon Valley startups and investment banks as well).

Sanjay and I spent quite a bit of time just talking about ideas and eventually decided to join forces. My Lucid Ventures incubator, along with Sanjay, seed-funded the new venture and named it Live Matrix, to go after our mutual vision.

Soon after Sanjay joined we were fortunate to be joined by our two highly experienced colleagues, Edgar Fereira (formerly VP of data for TV Guide Data and TV Guide Online) and Tobias Batton (serial entrepreneur, product manager, game designer). Then others joined around us.

Eventually we formed a small (but awesome) startup team and began working on a prototype and eventually an alpha. We debuted a closed beta preview at TechCrunch Disrupt last spring and received enthusiastic reviews. Now, today, we are releasing our public beta.

Read the full press release here.

I hope you like what we’ve created so far. But please note it is still a BETA. We are interested in your feedback and we already have a lot of feedback from our private beta. Here are some of the ideas we are working on for our next few releases:

• The Number One request we have received so far is to make it easier and faster for people to find events that would interest them. So for the remainder of the year one of our big priorities will be to add in more personalization and recommendations.
• We’re also working on new UI concepts, including some more ways to view the schedule of the Web.
• And we’re going to make it easier and faster for you to add events to Live Matrix — we’ll be launching improvements to our publisher tools section, as well more ways for people to suggest events for us to list.
• And we also plan to add new categories of events — for examples, Business, Technology, Games, and more.

So stay tuned! Live Matrix is just getting started. But this could be the start of something big.

ps. Here’s a screencast with a quick tour of Live Matrix

Live Matrix Demo from Doug Freeman on Vimeo.

Kate Ray has done a terrific job illustrating and explaining Web 3.0 and the Semantic Web in her new documentary. She interviews, Tim Berners-Lee, Clay Shirky, me, and many others. If you’re interested in where the Web is headed, and the challenges and opportunities ahead, then you should watch this, and share it too!

I have started blogging about a new concept that I call The Scheduled Web. The Scheduled Web is the next evolution of the Real-Time Web, in which it will become possible to actually navigate the time dimension of the Web more productively.

There is a popular misconception that on-demand content, such as archived video, is more valuable than live content. But in fact, this may not be the case.

Live content has built-in perishability that makes it potentially more valuable than on-demand content – if relevant audiences can find it while it is live. If a piece of high-demand content is only live for a short period of time it can attract more traffic in less time, provided that people who would want to participate interactively (or even transactively) in it are notified beforehand.

More demand in less time translates to higher advertising revenues, or higher prices in time-based sales like auctions. A series of high-demand live events could actually earn more revenues than a series of on-demand content releases in any given unit of time.

A live event is only live for some limited period of time, after which even though it may later be available in archived form, the event is finished, it is no longer a live event. If you want to get the live experience and be able to actually participate in a live event, you have to be there. It isn’t the same to watch it after the fact. And in some cases, for example auctions, sales, games, contests and chats, if you miss the event you can’t participate and may not even be able to access an archived version (if you even wanted to).

Live events are the best of both worlds for several reasons:

1. They have extra perishability because they are live, giving people a stronger incentive to participate synchronously when they are actually happening. Furthermore, if a live event is also interactive in some way, it is even more valuable to those who are present. A good example of this is American Idol, where for instance, the audience can participate in the voting process that selects finalists. Interactivity makes the show more engaging and gives viewers a sense of ownership and personal investment in the content.

2. Live events can also be archived and made available on-demand, as well. The key to getting this double-layer of value out of live events is to schedule them so that they can be found before or while they are actually live. This amplifies the initial demand and attendance to the event, and also provides any archived version that follows an added social virality.

At Live Matrix we believe it is incorrect to assume that the television model carries over directly to the Web. The Web is an entirely different medium because it is two-way, interactive, both synchronous and asynchronous, and distribution is open to anyone and portable across any device. Television over the Web is going to be different than TV on cable and satellite networks. The fact that consumers can consume Web video content asynchronously is a plus, but it doesn’t obviate the need or opportunity for live synchronous content on the Web. In fact, for any event that requires or even wants to leverage interactivity, live synchronous attendance by audience members is a key part of the experience.

There are many use-cases where live synchronous content consumption cannot be replaced by asynchronous content consumption — for example a live chat, or a time-limited sale or auction, or a multiplayer live game. Even in the case of video and audio there are many cases where live synchronous content is more valuable than asynchronous on-demand content. For example who wants to watch the Superbowl months after the game is over? Who really wants to watch a major presidential address or a press conference weeks later? Who wants to watch video of election coverage months after it’s decided? These kinds of “timely” events are live by their nature, and part of the value of consuming the content is the act of doing it in a timely manner.

The value of live interactive content begins to become even more clear as on-demand content that is originally streamed live has the ability to generate more revenues over its lifetime than simply recorded, on-demand content alone. The Scheduled Web will thus even improve traffic and revenues for on-demand content, if that content can be initiated as live events, or at least paired with them in some way.

The value of the Scheduled Web will be realized as not simply a schedule of video content, but of all scheduled events of any type that take place on the Internet. While much of this content is valuable both when it initially goes live and on an ongoing basis as on-demand content after the fact, there is also a lot of content in Live Matrix that will be inherently and necessarily more valuable when it is live, such as sales and auctions or games.

In addition there is a new category of “exclusively live” online events that we may see emerge in 2011. These events will be one-time events, with no archived copies after they finish. They may be high-profile events where attendance requires paid admission for example. They will be marketed as special experiences – where not only do you have to be there to experience them, but where being there has special advantages, like being able to interact with others who are there and perhaps with the performers or celebrities involved as well. Some events may also offer backstage passes, or special break-out sessions as well.

For events like these — where the only value created is during the event’s live run — discovery must happen prior to or during the event for participation to take place. For these, the Scheduled Web is absolutely essential.

If 2010 was the year of the Real-Time Web, then 2011 is going to be the year that it evolves into the Scheduled Web.

The Real-Time Web happens in the now: it is spontaneous, overwhelming, and disorganized. Things just happen unpredictably and nobody really knows what to expect or what will happen when.

The Real-Time Web is something of a misnomer, however, because usually it’s not real-time at all –  it’s after-the-fact. Most people find out about things that happened on the Real-Time Web after they happen, or, if they are lucky, when they happen. There is no way to know what is going to happen before it happens; there is no way to prepare or ensure that you will be online when something happens on the Real-Time Web. It’s entirely hit-or-miss.

If we are going to truly realize the Real-Time Web vision, then “time” needs to be the primary focus. So far, the Real-Time Web has mainly just been about simultaneity and speed – for example how quickly people on Twitter can respond to an event in the real world such as the Haiti Earthquake or the Oscars.

This obsession with the present is a sign of the times, but it is also a form of collective myopia — the Real-Time Web really doesn’t include the past or the future – it exists in a kind of perpetual now. To put the “time” into Real-Time, we need to  provide a way to see the past, present and the future Real-Time Web at once.  For example, we need a way to search and browse the past, present, and the future of a stream – what happened, what is happening, and what is scheduled to happen in the future. And this is where what I am calling The Scheduled Web comes in. It’s the next step for the Real-Time Web.

Defining the Scheduled Web

With the Scheduled Web things will start to make sense again. There will be a return of some semblance of order thanks to schedule metadata that enables people (and software) to find out about upcoming things on the Web that matter to them, before they happen, and to find out about past things that matter, after they happen.

The Scheduled Web is a Web that has a schedule, or many schedules, which exist in some commonly accessible, open format. These schedules should be searchable, linkable, shareable, interactive, collaborative, and discoverable. And they should be able to apply to anything — not just video, but any kind of content or activity online.

Why is this needed? Well consider this example. Imagine if there was no TV Guide on digital television. How would you navigate the constantly changing programming of more than 1000 digital TV channels without an interactive program guide (IPG)? It would be extremely difficult to find shows in a timely manner. According to clickstream data from television set-top boxes, about 10% of all time spent watching TV is spent in the IPG environment. And that is not even counting additional time-spent in on-demand guidance interfaces on DVRs. The point here is that guidance is key when you have lots of streams of content happening over time.

Now extend this same problem to the Web where there are literally millions of things happening every minute. These streams of content are not just limited to video. There are myriad types of real-time streams, everything from sales, auctions, and chats, to product launches, games, and audio, to streams of RSS feeds, Web pages appearing on Web sites, photos appearing on photo sites, software releases, announcements, etc.

Without some kind of guidance it is simply impossible to navigate the firehose of live online content streams on the Web efficiently. This firehose is too much to cope with in the present moment, let alone the past, or the future. This is what the Scheduled Web will solve.

By giving people a way to see into the past, present and future of the Real-Time Web, the Scheduled Web will enable the REAL Real-Time Web to be truly actualized. People will be able to know and plan in advance to actually be online when live events they care about take place.

Instead of missing that cool live Web concert or that auction for your favorite brand of shoes, simply because you didn’t know about it beforehand, you will be able to discover it in advance, RSVP, and get reminded before it starts — so you can be there and participate in the experience, right as it happens.

We are just beginning to see the emergence of the Scheduled Web. Two new examples of startups that are at work in the space are Clicker and Live Matrix.

• Clicker, a site that mainly provides on-demand video clips of past TV episodes, this week launched a schedule for live video streams on the Web.

• Live Matrix (my new startup), is soon to launch a schedule for all types of online events, not just video streams.

Some people have compared Live Matrix to Clicker, however this is not a wholly accurate comparison. We have very different, although  intersecting, goals.

While Clicker is an interesting play to compete with TV Guide and companies like Hulu, Live Matrix is creating a broader index of all the events taking place across the Scheduled Web, not just video/TV content events.

The insight behind Live Matrix is that there is much more to the Scheduled Web than video and TV content. The Web is not just about TV or video – it is about many different kinds of content.

Applying a TV metaphor to the Web is like trying to apply a print metaphor to tablet computing. While print has many positive qualities, tablet devices should not be limited just to text should they? Likewise, while the TV metaphor has advantages, it doesn’t make sense to limit the experience of time or scheduled content on the Web just to video.

With this in mind, while Live Matrix includes scheduled live video streams, we view video and TV type content as just one of many different types of scheduled Web content that matter.

For example, Live Matrix also includes online shopping events like sales and auctions, which comprise an enormous segment of the Scheduled Web. As an illustration eBay alone lists around 10 million scheduled auctions and sales each day! Live Matrix also includes scheduling metadata for many other kinds of content — online games, online chats, online audio, and more.

Live Matrix is building something quite a bit broader than current narrow conceptions of the Real-Time Web, or the narrow metaphor of TV on the Web. We are creating a way to navigate and search the full time dimension of the Web, we are building the schedule of the Web.

This will become a valuable, even essential, layer of metadata that just about every application, service and Internet surfer will make use of every day. Because after all, life happens in time and so does the Web. By adding metadata about time to the Web, Live Matrix will help make the Web – and particularly the Real-Time Web – easier to navigate.

Online vs. Offline Events

One of the key rules of Live Matrix is that, to be included in our schedule, an event must be consumable on-line. This means that it must be possible to access and participate in an event on an Internet-connected device.

Live Matrix is not a schedule of offline events or events that cannot be consumed or participated in using Internet-connected devices.

We made this rule because we believe that in the near-future almost everything interesting will, in fact, be consumable online, even if it has an offline component to it. We want to focus attention on those events which can be consumed on Internet-connected devices, so that if you have a connected device you can know that everything in Live Matrix can be accessed directly on your device. You don’t have to get in your car and drive to some physical venue, you don’t have to leave the Internet and go to some other device and network (like a TV and cable network).

Note the shift in emphasis here: We believe that the center of an increasing number of events is going to be online, and the offline world is going to increasingly become more peripheral.

For example, if a retail sale generates more revenues from online purchases than physical in-store purchases, the center of the sale is really on-line and the physical store becomes peripheral. Similarly, if a live concert has 30,000 audience members in a physical stadium but 10,000,000 people attending it online, the bulk of the concert is in fact online. This is already starting to happen.

For example, the recent Youtube concert featuring U2 had 10 million live streams – that’s up to 10 million live people in the audience at one time, making it possibly the largest online concert in history; it’s certainly a lot more people than any physical stadium could accommodate. Similarly, online venues like Second Life and World of Warcraft can accommodate thousands of players interacting in the same virtual spaces – not only do these spaces not even have a physical analogue (they exist only in virtual space), but there are no physical spaces that could accommodate such large games. These are examples of how online events may start to eclipse offline events.

I’m not saying this trend is good or bad; I’m simply stating a fact of our changing participatory culture. The world is going increasingly online and with this shift the center of our lives is going increasingly online, as well. It is this insight that gave my co-founder, Sanjay Reddy, and I, the inspiration to start Live Matrix, and to begin building what we hope will be the backbone of the Scheduled Web.

The emerging realtime Web is not only going to speed up the Web and our lives, it is going to bring about a kind of awakening of our collective Global Brain. It’s going to change how many things happen on online, but it’s also going to change how we see and understand what the Web is doing. By speeding up the Web, it will cause processes that used to take weeks or months to unfold online, to happen in days or even minutes. And this will bring these processes to the human-scale — to the scale of our human “now” — making it possible for us to be aware of larger collective processes than before. We have until now been watching the Web in slow motion. As it speeds up, we will begin to see and understand what’s taking place on the Web in a whole new way.

This process of of quickening is part of a larger trend which I and others call “Nowism.” You can read more of my thoughts about Nowism here. Nowism is an orientation that is gaining momentum and will help to shape this decade, and in particular, how the Web unfolds. It is the idea that the present-timeframe (“the now”) is getting more important, shorter and also more information-rich. As this happens our civilization is becoming more focused on the now, and less focused on past or the future. Simply keeping up with the present is becoming an all-consuming challenge: Both a threat and an opportunity.

The realtime Web –  what I call “The Stream”  (see “Welcome to the Stream”) — is changing the unit of now. It’s making it shorter. The now is the span of time which we have to be aware of to be effective our work and lives, and it is getting shorter. On a personal level the now is getting shorter and denser — more information and change is packed into shorter spans of time; a single minute on Twitter is overflowing with potentially relevant messages and links. In business as well, the now is getting shorter and denser — it used to be about the size of a fiscal quarter, then it became a month, then a week, then a day, and now it is probably about half a day in span. Soon it will be just a few hours.

To keep up with what is going on we have to check in with the world in at least half-day chunks. Important news breaks about once or twice a day. Trends on Twitter take about a day to develop too. So basically, you can afford to just check  the news and the real-time Web once or twice a day and still get by. But that’s going to change.  As the now gets shorter, we’ll have to check in more frequently to keep abreast of change. As the Stream picks up speed in the middle of this decade, to remain competitive will require near-constant monitoring — we will have to always be connected to, and watching, the real-time Web and our personal streams. Being offline at all will risk missing out on big important trends, threats and opportunities that emerge and develop within minutes or hours. But nobody is capable of tracking the Stream all 24/7 — we must at least take breaks to eat and sleep. And this is a problem.

Big Changes to the Web Coming Soon…

With Nowism comes a faster Web, and this will lead to big changes in how we do various activities on the Web:

• We will spend less time searching. Nowism pushes us to find better alternatives to search, or to eliminate search entirely, because people don’t have time to search anymore. We need tools that do the searching for us and that help with decision support so we don’t have to spend so much of our scarce time doing that. See my article on “Eliminating the Need for Search — Help Engines” for more about that.
• Monitoring (not searching) the real-time stream becomes more important. We need to stay constantly vigilant about what’s happening, what’s trending. We need to be alerted of the important stuff (to us), and we need a way to filter out what’s not important to us. Probably a filter based on influence of people and tweets, and/or time dynamics of memes will be necessary. Monitoring the real-time stream effectively is different from searching it. I see more value in real-time monitoring than realtime search — I haven’t seen any monitoring tools for Twitter that are smart enough to give me just the content I want yet. There’s a real business opportunity there.
• The return of agents. Intelligent agents are going to come back. To monitor the realtime Web effectively each of us will need online intelligent agents that can help us — because we don’t have time, and even if we did, there’s just too much information to sift through.
• Influence becomes more important than relevance. Advertisers and marketers will look for the most influential parties (individuals or groups) on Twitter and other social media to connect with and work through. But to do this there has to be an effective way to measure influence. One service that’s providing a solution for this (which I’ve angel invested in and advise) is Klout.com – they measure influence per person per topic. I think that’s a good start.
• Filtering content by influence. We also will need a way to find the most influential content. Influential content could be the content most RT’d or most RT’d by most influential people. It would be much less noisy to be able to see only the more influential tweets of people I follow. If a tweet gets RT’d a lot, or is RT’d by really influential people, then I want to see it. If not, then only if it’s really important (based on some rule). This will be the only way to cope with the information overload of the real-time Web and keep up with it effectively. I don’t know of anyone providing a service for this yet. It’s a business opportunity.
• Nowness as a measure of value of content. We will need a new form of ranking of results by “nowness” – how timely they are now. So for example, in real-time search engines we shouldn’t rank results merely by how recent they are, but also by how timely, influential, and “hot” they are now. See my article from years ago on “A Physics of Ideas” for more about that. Real-time search companies should think of themselves as real-time monitoring companies — that’s what they are really going to be used for in the end. Only the real-time search ventures that think of themselves this way are going to survive the conceptual paradigm shift that the realtime Web is bringing about. In a realtime context, search is actually too late — once something has happened in the past it really is not that important anymore –what matters is current awareness: discovering the trends NOW. To do that one has to analyze the present, and the very recent past, much more than searching the longer term past. The focus has to be on real-time or near-real-time analytics, statistical analysis, topic and trend detection, prediction, filtering and alerting. Not search.
• New ways to understand and navigate the now. We will need a way to visualize and navigate the now. I’m helping to incubate a stealth startup venture, Live Matrix, that is working on that. It hasn’t launched yet. It’s cool stuff. More on that in the future when they launch.
• New tools for browsing the Stream. New tools will emerge for making the realtime Web more compelling and smarter. I’m working on incubating some new stealth startups in this area as well. They’re very early-stage so can’t say more about them yet.
• The merger of semantics with the realtime Web. We need to make the realtime Web semantic — as well as the rest of the Web — in order to make it easier for software to make sense of it for us. This is the best approach to increasing the signal-to-noise ratio of content we have to look at whether searching or monitoring stuff. The Semantic Web standars of the W3C are key to this. I’ve written a long manifesto on this in “Minding The Planet: The Meaning and Future of the Semantic Web” if you’re really interested in that topic.

Faster Leads to Smarter

As the realtime web unfolds and speeds up, I think it will also have a big impact on what some people call “The Global Brain.” The Global Brain has always existed, but in recent times it has been experiencing a series of major upgrades — particularly around how connected, affordable, accessible and fast it is. First we got phone and faxes, then the Internet, the PC and the Web, and now the real-time Web and the Semantic Web. All of these recent changes are making the Global Brain faster, more richly interconnected. And this makes it smarter. For more about my thoughts on the Global Brain, see these two talks:

• My detailed History and Future of the Global Brain given at the Singularity Summit.
• A talk on the emerging Global Brain and human-machine cybernetic superorganism, with specific focus on what it means for media companies, from the GRID Conference.

What’s most interesting to me is that as the rate of communication and messaging on the Web approaches near-real time, we may see a kind of phase change take place – a much smarter Global Brain will sort of begin to appear out of the chaos. In other words, the speed of collective thinking is as important to the complexity or sophistication of collective thinking, in making the Global Brain significantly more intelligent. In other words, I’m proposing that there is a sort of critical speed of collective thinking, before which the Global Brain seems like just a crowd of actors chaotically flocking around memes, and after which the Global Brain makes big leaps — instead of seeming like a chaotic crowd, it starts to look more like an organized group around certain activitities — it is able to respond to change faster, and optimize and even do things collectively more productively than a random crowd could.

This is kind of like film, or animation. When you watch a movie or animation you are really watching a rapid series of frames. This gives the illusion of there being cohesive, continuous characters, things and worlds in the movie — but really they aren’t there at all, it’s just an illusion — our brains put these scenes together and start to recognize and follow higher order patterns. A certain shape appears to maintain itself and move around relative to other shapes, and we name it with a certain label — but there isn’t really something there, let alone something moving or interacting — there are just frames flicking by rapidly . It turns out that after a critical frame rate (around 20 to 60 frames per second) the human brain stops seeing individual frames and starts seeing a continuous movie. When you start flipping pages fast enough it appears to be a coherent animation and then we start seeing things “moving within the sequence” of frames. In the same way, as the unit of time of (aka the speed) of the real-time Web increases, its behavior will start to seem more continuous and smarter — we won’t see separate chunks of time or messages, we’ll see intelligent continuous collective thinking and adaptation processes.

In other words, as the Web gets faster, we’ll start to see processes emerge within it that appear to be cohesive intelligent collective entities in their own right. There won’t really be any actual entities there that we can isolate, but when we watch the patterns on the Web it will appear as if such entities are there. This is basically what is happening at every level of scale — even in the real world. There really isn’t anything there that we can find — everything is divisible down to the quantum level and probably beyond — but over time our brains seem to recognize and label patterns as discrete “things.” This is what will happen across the Web as well. For example, a certain meme (such as a fad or a movement) may become a “thing” in it’s own right, a kind of entity that seemingly takes on a life of its own and seems to be doing something. Similarly certain groups or social networks or activities they engage in may seem to be intelligent entities in their own rights.

This is an illusion in that there really are no entities there, they are just collections of parts that themselves can be broken down into more parts, and no final entities can be found. However, nonethless, they will seem like intelligent entities when not analyzed in detail. In addition, the behavior of these chaotic systems may resist reduction — they may not even be understandable and their behavior may not be predictable through a purely reductionist approach — it may be that they react to their own internal state and their environments virtually in real-time, making it difficult to take a top-down or bottom-up view of what they are doing. In a realtime world, change happens in every direction.

As the Web gets faster, the patterns that are taking place across it will start to become more animated. Big processes that used to take months or years to happen will happen in minutes or hours. As this comes about we will begin to see larger patterns than before, and they will start to make more sense to us — they will emerge out of the mists of time so to speak, and become visible to us on our human timescale — the timescale of our human-level “now. As a result, we will become more aware of higher order dynamics taking place on the real-time Web, and we will begin to participate in and adapt to those dynamics, making those dynamics in turn even smarter. (For more on my thoughts about how the Global Brain gets smarter, see:  “How to Build the Global Mind.”)

See Part II: “Will The Web Become Conscious?” if you want to dig further into the thorny philosophical and scientific issues that this brings up…

This is the story of Twine.com — our early research (with never before seen screenshots of our early semantic desktop work), and our evolution from Twine 1.0 towards Twine 2.0 (“T2″) which is focused on semantic search.

This is a talk I have given many times, on the past, present and future evolution of the Web, and particularly the Semantic Web.

The next generation of Web search is coming sooner than expected. And with it we will see several shifts in the way people search, and the way major search engines provide search functionality to consumers.

Web 1.0, the first decade of the Web (1989 – 1999), was characterized by a distinctly desktop-like search paradigm. The overriding idea was that the Web is a collection of documents, not unlike the folder tree on the desktop, that must be searched and ranked hierarchically. Relevancy was considered to be how closely a document matched a given query string.

Web 2.0, the second decade of the Web (1999 – 2009), ushered in the beginnings of a shift towards social search. In particular blogging tools, social bookmarking tools, social networks, social media sites, and microblogging services began to organize the Web around people and their relationships. This added the beginnings of a primitive “web of trust” to the search repertoire, enabling search engines to begin to take the social value of content (as evidences by discussions, ratings, sharing, linking, referrals, etc.) as an additional measurment in the relevancy equation. Those items which were both most relevant on a keyword level, and most relevant in the social graph (closer and/or more popular in the graph), were considered to be more relevant. Thus results could be ranked according to their social value — how many people in the community liked them and current activity level — as
well as by semantic relevancy measures.

In the coming third decade of the Web, Web 3.0 (2009 – 2019), there will be another shift in the search paradigm. This is a shift to from the past to the present, and from the social to the personal.

Established search engines like Google rank results primarily by keyword (semantic) relevancy. Social search engines rank results primarily by activity and social value (Digg, Twine 1.0, etc.). But the new search engines of the Web 3.0 era will also take into account two additional factors when determining relevancy: timeliness, and personalization.

Google returns the same results for everyone. But why should that be the case? In fact, when two different people search for the same information, they may want to get very different kinds of results. Someone who is a novice in a field may want beginner-level information to rank higher in the results than someone who is an expert. There may be a desire to emphasize things that are novel over things that have been seen before, or that have happened in the past — the more timely something is the more relevant it may be as well.

These two themes — present and personal — will define the next great search experience.

To accomplish this, we need to make progress on a number of fronts.

First of all, search engines need better ways to understand what content is, without having to do extensive computation. The best solution for this is to utilize metadata and the methods of the emerging semantic web.

Metadata reduces the need for computation in order to determine what content is about — it makes that explicit and machine-understandable. To the extent that machine-understandable metadata is added or generated for the Web, it will become more precisely searchable and productive for searchers.

This applies especially to the area of the real-time Web, where for example short “tweets” of content contain very little context to support good natural-language processing. There a little metadata can go a long way. In addition, of course metadata makes a dramatic difference in search of the larger non-real-time Web as well.

In addition to metadata, search engines need to modify their algorithms to be more personalized. Instead of a “one-size fits all” ranking for each query, the ranking may differ for different people depending on their varying interests and search histories.

Finally, to provide better search of the present, search has to become more realtime. To this end, rankings need to be developed that surface not only what just happened now, but what happened recently and is also trending upwards and/or of note. Realtime search has to be more than merely listing search results chronologically. There must be effective ways to filter the noise and surface what’s most important effectively. Social graph analysis is a key tool for doing this, but in
addition, powerful statistical analysis and new visualizations may also be required to make a compelling experience.

Sneak Preview of Siri – The Virtual Assistant that will Make Everyone Love the iPhone, Part 2: The Technical Stuff

In Part-One of this article on TechCrunch, I covered the emerging paradigm of Virtual Assistants and explored a first look at a new product in this category called Siri. In this article, Part-Two, I interview Tom Gruber, CTO of Siri, about the history, key ideas, and technical foundations of the product:

Nova Spivack: Can you give me a more precise definition of a Virtual Assistant?

Tom Gruber: A virtual personal assistant is a software system that

• Helps the user find or do something (focus on tasks, rather than information)
• Understands the user’s intent (interpreting language) and context (location, schedule, history)
• Works on the user’s behalf, orchestrating multiple services and information sources to help complete the task

In other words, an assistant helps me do things by understanding me and working for me. This may seem quite general, but it is a fundamental shift from the way the Internet works today. Portals, search engines, and web sites are helpful but they don’t do things for me – I have to use them as tools to do something, and I have to adapt to their ways of taking input.

Nova Spivack: Siri is hoping to kick-start the revival of the Virtual Assistant category, for the Web. This is an idea which has a rich history. What are some of the past examples that have influenced your thinking?

Tom Gruber: The idea of interacting with a computer via a conversational interface with an assistant has excited the imagination for some time.  Apple’s famous Knowledge Navigator video offered a compelling vision, in which a talking head agent helped a professional deal with schedules and access information on the net. The late Michael Dertouzos, head of MIT’s Computer Science Lab, wrote convincingly about the assistant metaphor as the natural way to interact with computers in his book “The Unfinished Revolution: Human-Centered Computers and What They Can Do For Us”.  These accounts of the future say that you should be able to talk to your computer in your own words, saying what you want to do, with the computer talking back to ask clarifying questions and explain results.  These are hallmarks of the Siri assistant.  Some of the elements of these visions
are beyond what Siri does, such as general reasoning about science in the Knowledge Navigator.  Or self-awareness a la Singularity.  But Siri is the real thing, using real AI technology, just made very practical on a small set of domains. The breakthrough is to bring this vision to a mainstream market, taking maximum advantage of the mobile context and internet service ecosystems.

Nova Spivack: Tell me about the CALO project, that Siri spun out from. (Disclosure: my company, Radar Networks, consulted to SRI in the early days on the CALO project, to provide assistance with Semantic Web development)

Tom Gruber: Siri has its roots in the DARPA CALO project (“Cognitive Agent that Learns and Organizes”) which was led by SRI. The goal of CALO was to develop AI technologies (dialog and natural language understanding,s understanding, machine learning, evidential and probabilistic reasoning, ontology and knowledge representation, planning, reasoning, service delegation) all integrated into a virtual
assistant that helps people do things.  It pushed the limits on machine learning and speech, and also showed the technical feasibility of a task-focused virtual assistant that uses knowledge of user context and multiple sources to help solve problems.

Siri is integrating, commercializing, scaling, and applying these technologies to a consumer-focused virtual assistant.  Siri was under development for several years during and after the CALO project at SRI. It was designed as an independent architecture, tightly integrating the best ideas from CALO but free of the constraints of a national distributed research project. The Siri.com team has been evolving and hardening the technology since January 2008.

Nova Spivack: What are primary aspects of Siri that you would say are “novel”?

Tom Gruber: The demands of the consumer internet focus — instant usability and robust interaction with the evolving web — has driven us to come up with some new innovations:

• A conversational interface that combines the best of speech and semantic language understanding with an interactive dialog that helps guide
  people toward saying what they want to do and getting it done. The
  conversational interface allows for much more interactivity that one-shot search style interfaces, which aids usability and improves intent understanding.  For example, if Siri didn’t quite hear what you said, or isn’t sure what you meant, it can ask for clarifying information.   For example, it can prompt on ambiguity: did you mean pizza restaurants in Chicago or Chicago-style pizza places near you? It can also make reasonable guesses based on context. Walking around with the phone at lunchtime, if the speech interpretation comes back with something garbled about food you probably meant “places to eat near my current location”. If this assumption isn’t right, it is easy to correct in a conversation.
• Semantic auto-complete – a combination of the familiar “autocomplete” interface of search boxes with a semantic and linguistic model of what might be worth saying. The so-called “semantic completion” makes it possible to rapidly state complex requests (Italian restaurants in the SOMA neighborhood of San Francisco that have tables available tonight) with just a few clicks. It’s sort of like the power of faceted search a la Kayak, but packaged in a clever command line style interface that works in small form factor and low bandwidth environments.
• Service delegation – Siri is particularly deep in technology for operationalizing a user’s intent into computational form, dispatching to multiple, heterogeneous services, gathering and integrating results, and presenting them back to the user as a set of solutions to their request.  In a restaurant selection task, for instance, Siri combines information from many different sources (local business directories, geospatial databases, restaurant guides, restaurant review sources, online reservation services, and the user’s own favorites) to show a set of candidates that meet the intent expressed in the user’s natural language request.

Nova Spivack: Why do you think Siri will succeed when other AI-inspired projects have failed to meet expectations?

Tom Gruber: In general my answer is that Siri is more focused. We can break this down into three areas of focus:

• Task focus. Siri is very focused on a bounded set of specific human tasks, like finding something to do, going out with friends, and getting around town.  This task focus allows it to have a very rich model of its domain of competence, which makes everything more tractable from language understanding to reasoning to service invocation and results presentation
  
• Structured data focus. The kinds of tasks that Siri is particularly good at involve semistructured data, usually on tasks involving multiple criteria and drawing from multiple sources.  For example, to help find a place to eat, user preferences for cuisine, price range, location, or even specific food items come into play.  Combining results from multiple sources requires
  reasoning about domain entity identity and the relative capabilities of different information providers.  These are hard problems of semantic
  information processing and integration that are difficult but feasible
  today using the latest AI technologies.
• Architecture focus. Siri is built from deep experience in integrating multiple advanced technologies into a platform designed expressly for virtual assistants. Siri co-founder Adam Cheyer was chief architect of the CALO project, and has applied a career of experience to design the platform of the Siri product. Leading the CALO project taught him a lot about what works and doesn’t when applying AI to build a virtual assistant. Adam and I also have rather unique experience in combining AI with intelligent interfaces and web-scale knowledge integration. The result is a “pure  play” dedicated architecture for virtual assistants, integrating all the components of intent understanding, service delegation, and dialog flow management. We have avoided the need to solve general AI problems by concentrating on only what is needed for a virtual assistant, and have chosen to begin with a
  finite set of vertical domains serving mobile use cases.

Nova Spivack: Why did you design Siri primarily for mobile devices, rather than Web browsers in general?

Tom Gruber: Rather than trying to be like a search engine to all the world’s information, Siri is going after mobile use cases where deep models of context (place, time, personal history) and limited form factors magnify the power of an intelligent interface.  The smaller the form factor, the more mobile the context,
the more limited the bandwidth : the more it is important that the interface make intelligent use of the user’s attention and the resources at hand.  In other words, “smaller needs to be smarter.”  And the benefits of being offered just the right level of detail or being prompted with just the right questions can make the difference between task completion or failure.  When you are on the go, you just don’t have time to wade through pages of links and disjoint interfaces, many of which are not suitable to mobile at all.

Nova Spivack: What language and platform is Siri written in?

Tom Gruber: Java, Javascript, and Objective C (for the iPhone)

Nova Spivack: What about the Semantic Web? Is Siri built with Semantic Web open-standards such as RDF and OWL, Sparql?

Tom Gruber: No, we connect to partners on the web using structured APIs, some of which do use the Semantic Web standards.  A site that exposes RDF usually has an API that is easy to deal with, which makes our life easier.  For instance, we use geonames.org as one of our geospatial information sources. It is a full-on Semantic
Web endpoint, and that makes it easy to deal with.  The more the API declares its data model, the more automated we can make our coupling to it.

Nova Spivack: Siri seems smart, at least about the kinds of tasks it was designed for. How is the knowledge represented in Siri – is it an ontology or something else?

Tom Gruber: Siri’s knowledge is represented in a unified modeling system that combines ontologies, inference networks, pattern matching agents, dictionaries, and dialog models.  As much as possible we represent things declaratively (i.e., as data in models, not lines of code).  This is a tried and true best practice for complex AI systems.  This makes the whole system more robust and scalable, and the development process more agile.  It also helps with reasoning and learning, since Siri can look at what it knows and think about similarities and generalizations at a semantic level.

Nova Spivack: Will Siri be part of the Semantic Web, or at least the open linked data Web (by making open API’s, sharing of linked data, RDF, available, etc.)?

Tom Gruber: Siri isn’t a source of data, so it doesn’t expose data using Semantic Web standards.  In the Semantic Web ecosystem, it is doing something like the vision of a semantic desktop – an intelligent interface that knows about user needs
and sources of information to meet those needs, and intermediates.  The original Semantic Web article in Scientific American included use cases that an assistant would do (check calendars, look for things based on multiple structured criteria, route planning, etc.).  The Semantic Web vision focused on exposing the structured data, but it assumes APIs that can do transactions on the data.  For example, if a virtual assistant wants to schedule a dinner it needs more than the information
about the free/busy schedules of participants, it needs API access to their calendars with appropriate credentials, ways of communicating with the participants via APIs to their email/sms/phone, and so forth. Siri is building on the ecosystem of APIs, which are better if they declare the meaning of the data in and out via ontologies.  That is the original purpose of ontologies-as-specification that I promoted in the
1990s – to help specify how to interact with these agents via knowledge-level APIs.

Siri does, however, benefit greatly from standards for talking about space and time, identity (of people, places, and things), and authentication.  As I called for in my Semantic Web talk in 2007, there is no reason we should be string matching on city names, business names, user names, etc.

All players near the user in the ecommerce value chain get better when the information that the users need can be unambiguously identified, compared, and combined. Legitimate service providers on the supply end of the value chain also benefit, because structured data is harder to scam than text.  So if some service provider offers a multi-criteria decision making service, say, to help make a product purchase in some domain, it is much easier to do fraud detection when the product instances, features, prices, and transaction availability information are all structured data.

Nova Spivack: Siri appears to be able to handle requests in natural language. How good is the natural language processing (NLP) behind it? How have you made it better than other NLP?

Tom Gruber: Siri’s top line measure of success is task completion (not relevance).  A subtask is intent recognition, and subtask of that is NLP.  Speech is another element, which couples to NLP and adds its own issues.  In this context, Siri’s NLP is “pretty darn good” — if the user is talking about something in Siri’s domains of competence, its intent understanding is right the vast majority of the time, even in the face of noise from speech, single finger typing, and bad habits from too much keywordese.  All NLP is tuned for some class of natural language, and Siri’s is tuned for things that people might want to say when talking to a virtual assistant on their phone. We evaluate against a corpus, but I don’tknow how it would compare to standard message and news corpuses using by the NLP research community.

Nova Spivack: Did you develop your own speech interface, or are you using third-party system for that? How good is it? Is it battle-tested?

Tom Gruber: We use third party speech systems, and are architected so we can swap them out and experiment. The one we are currently using has millions of users and continuously updates its models based on usage.

Nova Spivack: Will Siri be able to talk back to users at any point?

Tom Gruber: It could use speech synthesis for output, for the appropriate contexts.  I have a long standing interest in this, as my early graduate work was in communication prosthesis. In the current mobile internet world, however, iPhone-sized screens and 3G networks make it possible to do so more much than read menu items over the phone.  For the blind, embedded appliances, and other applications it would make sense to give Siri voice output.

Nova Spivack: Can you give me more examples of how the NLP in Siri works?

Tom Gruber: Sure, here’s an example, published in the Technology Review, that illustrates what’s going on in a typical dialogue with Siri. (Click link to view the table)

Nova Spivack: How personalized does Siri get – will it recommend different things to me depending on where I am when I ask, and/or what I’ve done in the past? Does it learn?

Tom Gruber: Siri does learn in simple ways today, and it will get more sophisticated with time.  As you said, Siri is already personalized based on immediate context, conversational history, and personal information such as where you live.  Siri doesn’t forget things from request to request, as do stateless systems like search engines. It always considers the user model along with the domain and task models when coming up with results.  The evolution in learning comes as users have a history with Siri, which gives it achance to make some generalizations about preferences.  There is a natural progression with virtual assistants from doing exactly what they are asked, to making recommendations based on assumptions about intent and preference. That is the curve we will explore with experience.

Nova Spivack: How does Siri know what is in various external services – are you mining and doing extraction on their data, or is it all just real-time API calls?

Tom Gruber: For its current domains Siri uses dozens of APIs, and connects to them in both realtime access and batch data synchronization modes.  Siri knows about the data because we (humans) explicitly model what is in those sources.  With declarative representations of data and API capabilities, Siri can reason about the various capabilities of its sources at run time to figure out which combination would best serve the current user request.  For sources that do not have nice APIs or expose data using standards like the Semantic Web, we can draw on a value chain of players that do extract structure by data mining and exposing APIs via scraping.

Nova Spivack: Thank you for the information, Siri might actually make me like the iPhone enough to start using one again.

Tom Gruber: Thank you, Nova, it’s a pleasure to discuss this with someone who really gets the technology and larger issues. I hope Siri does get you to use that iPhone again. But remember, Siri is just starting out and will sometimes say silly things. It’s easy to project intelligence onto an assistant, but Siri isn’t going to pass the Turing Test. It’s just a simpler, smarter way to do what you already want to do. It will be interesting to see how this space evolves, how people will come to understand what to expect from the little personal assistant in their pocket.

I’ve written a new article about how content distribution has evolved, and where it is heading. It’s published here: http://www.siliconangle.com/social-media/content-distribution-is-changing-again/.

UPDATE: There’s already a lot of good discussion going on around this post in my public twine.

I’ve been writing about a new trend that I call “interest networking” for a while now. But I wanted to take the opportunity before the public launch of Twine on Tuesday (tomorrow) to reflect on the state of this new category of applications, which I think is quickly reaching its tipping point. The concept is starting to catch on as people reach for more depth around their online interactions.

In fact – that’s the ultimate value proposition of interest networks – they move us beyond the super poke and towards something more meaningful. In the long-term view, interest networks are about building a global knowledge commons. But in the short term, the difference between social networks and interest networks is a lot like the difference between fast food and a home-cooked meal – interest networks are all about substance.

At a time when social media fatigue is setting in, the news cycle is growing shorter and shorter, and the world is delivered to us in soundbytes and catchphrases, we crave substance. We go to great lengths in pursuit of substance. Interest networks solve this problem – they deliver substance.t

So, what is an interest network?

In short, if a social network is about who you are interested in, an interest network is about what you are interested in. It’s the logical next step.

Twine for example, is an interest network that helps you share information with friends, family, colleagues and groups, based on mutual interests. Individual “twines” are created for content around specific subjects. This content might include bookmarks, videos, photos, articles, e-mails, notes or even documents. Twines may be public or private and can serve individuals, small groups or even very large groups of members.

I have also written quite a bit about the Semantic Web and the Semantic Graph, and Tim Berners-Lee has recently started talking about what he calls the GGG (Giant Global Graph). Tim and I are in agreement that social networks merely articulate the relationships between people. Social networks do not surface the equally, if not more important, relationships between people and places, places and organizations, places and other places, organization and other organizations, organization and events, documents and documents, and so on.

This is where interest networks come in. It’s still early days to be clear, but interest networks are operating on the premise of tapping into a multi–dimensional graph that manifests the complexity and substance of our world, and delivers the best of that world to you, every day.

We’re seeing more and more companies think about how to capitalize on this trend. There are suddenly (it seems, but this category has been building for many months) lots of different services that can be viewed as interest networks in one way or another, and here are some examples:

• Twine (my site)
• Friendfeed
• Strands
• Intuu
• SocialMedian
• Zimesh
• Popego
• YourVersion

What all of these interest networks have in common is some sort of a bottom-up, user-driven crawl of the Web, which is the way that I’ve described Twine when we get the question about how we propose to index the entire Web (the answer: we don’t.

We let our users tell us what they’re most interested in, and we follow their lead).

Most interest networks exhibit the following characteristics as well:

• They have some sort of bookmarking/submission/markup function to store and map data (often using existing metaphors, even if what’s under the hood is new)
• They also have some sort of social sharing function to provide the network benefit (this isn’t exclusive to interest networks, obviously, but it is characteristic)
• And in most cases, interest networks look to add some sort of “smarts” or “recommendations” capability to the mix (that is, you get more out than you put in)

This last bullet point is where I see next-generation interest networks really providing the most benefit over social bookmarking tools, wikis, collaboration suites and pure social networks of one kind or another.

To that end, we think that Twine is the first of a new breed of intelligent applications that really get to know you better and better over time – and that the more you use Twine, the more useful it will become. Adding your content to Twine is an investment in the future of your data, and in the future of your interests.

At first Twine begins to enrich your data with semantic tags and links to related content via our recommendations engine that learns over time. Twine also crawls any links it sees in your content and gathers related content for you automatically – adding it to your personal or group search engine for you, and further fleshing out the semantic graph of your interests which in turn results in even more relevant recommendations.

The point here is that adding content to Twine, or other next-generation interest networks, should result in increasing returns. That’s a key characteristic, in fact, of the interest networks of the future – the idea that the ratio of work (input) to utility (output) has no established ceiling.

Another key characteristic of interest networks may be in how they monetize. Instead of being advertising-driven, I think they will focus more on a marketing paradigm. They will be to marketing what search engines were to advertising. For example, Twine will be monetizing our rich model of individual and group interests, using our recommendation engine. When we roll this capability out in 2009, we will deliver extremely relevant, useful content, products and offers directly to users who have demonstrated they are really interested in such information, according to their established and ongoing preferences.

6 months ago, you could not really prove that “interest networking” was a trend, and certainly it wasn’t a clearly defined space. It was just an idea, and a goal. But like I said, I think that we’re at a tipping point, where the technology is getting to a point at which we can deliver greater substance to the user, and where the culture is starting to crave exactly this kind of service as a way of making the Web meaningful again.

I think that interest networks are a huge market opportunity for many startups thinking about what the future of the Web will be like, and I think that we’ll start to see the term used more and more widely. We may even start to see some attention from analysts — Carla, Jeremiah, and others, are you listening?

Now, I obviously think that Twine is THE interest network of choice. After all we helped to define the category, and we’re using the Semantic Web to do it. There’s a lot of potential in our engine and our application, and the growing community of passionate users we’ve attracted.

Our 1.0 release really focuses on UE/usability, which was a huge goal for us based on user feedback from our private beta, which began in March of this year. I’ll do another post soon talking about what’s new in Twine. But our TOS (time on site) at 6 minutes/user (all time) and 12 minutes/user (over the last month) is something that the team here is most proud of – it tells us that Twine is sticky, and that “the dogs are eating the dog food.”

Now that anyone can join, it will be fun and gratifying to watch Twine grow.

Still, there is a lot more to come, and in 2009 our focus is going to shift back to extending our Semantic Web platform and turning on more of the next-generation intelligence that we’ve been building along the way. We’re going to take interest networking to a whole new level.

Stay tuned!

I’ve posted a link to a video of my best talk — given at the GRID ’08 Conference in Stockholm this summer. It’s about the growth of collective intelligence and the Semantic Web, and the future and role the media. Read more and get the video here. Enjoy!

I’ve posted a new article in my public twine about how we are moving from the World Wide Web to the Web Wide World. It’s about how the Web is spreading into the physical world, and what this means.

Video from my panel at DEMO Fall ’08 on the Future of the Web is now available.

I moderated the panel, and our panelists were:

Howard Bloom, Author, The Evolution of Mass Mind from the Big Bang to the 21st Century

Peter Norvig, Director of Research, Google Inc.

Jon Udell, Evangelist, Microsoft Corporation

Prabhakar Raghavan, PhD, Head of Research and Search Strategy, Yahoo! Inc.

The panel was excellent, with many DEMO attendees saying it was the best panel they had ever seen at DEMO.

Many new and revealing insights were provided by our excellent panelists. I was particularly interested in the different ways that Google and Yahoo describe what they are working on. They covered lots of new and interesting information about their thinking. Howard Bloom added fascinating comments about the big picture and John Udell helped to speak about Microsoft’s longer-term views as well.

Enjoy!!!

Here is the full video of my talk on the Semantic Web at The Next Web 2008 Conference. Thanks to Boris and the NextWeb gang!

I have been thinking a lot about social networks lately, and why there are so many of them, and what will happen in that space.

Today I had what I think is a "big realization" about this.

Everyone, including myself, seems to think that there is only room for one big social network, and it looks like Facebook is winning that race. But what if that assumption is simply wrong from the start?

What if social networks are more like automobile brands? In other words, there can, will and should be many competing brands in the space?

Social networks no longer compete on terms of who has what members. All my friends are in pretty much every major social network.

I also don’t need more than one social network, for the same reason — my friends are all in all of them. How many different ways do I need to reach the same set of people? I only need one.

But the Big Realization is that no social network satisfies all types of users. Some people are more at home in a place like LinkedIn than they are in Facebook, for example. Others prefer MySpace.  There are always going to be different social networks catering to the common types of people (different age groups, different personalities, different industries, different lifestyles, etc.).

The Big Realization implies that all the social networks are going to be able to interoperate eventually, just like almost all email clients and servers do today. Email didn’t begin this way. There were different networks, different servers and different clients, and they didn’t all speak to each other. To communicate with certain people you had to use a certain email network, and/or a certain email program. Today almost all email systems interoperate directly or at least indirectly. The same thing is going to happen in the social networking space.

Today we see the first signs of this interoperability emerging as social networks open their APIs and enable increasing integration. Currently there is a competition going on to see which "open" social network can get the most people and sites to use it. But this is an illusion. It doesn’t matter who is dominant, there are always going to be alternative social networks, and the pressure to interoperate will grow until it happens. It is only a matter of time before they connect together.

I think this should be the greatest fear at companies like Facebook. For when it inevitably happens they will be on a level playing field competing for members with a lot of other companies large and small. Today Facebook and Google’s scale are advantages, but in a world of interoperability they may actually be disadvantages — they cannot adapt, change or innovate as fast as smaller, nimbler startups.

Thinking of social networks as if they were automotive brands also reveals interesting business opportunities. There are still several unowned opportunities in the space.

Myspace is like the car you have in high school. Probably not very expensive, probably used, probably a bit clunky. It’s fine if you are a kid driving around your hometown.

Facebook is more like the car you have in college. It has a lot of your junk in it, it is probably still not cutting edge, but its cooler and more powerful.

LinkedIn kind of feels like a commuter car to me. It’s just for business, not for pleasure or entertainment.

So who owns the "adult luxury sedan" category? Which one is the BMW of social networks?

Who owns the sportscar category? Which one is the Ferrari of social networks?

Who owns the entry-level commuter car category?

Who owns equivalent of the "family stationwagon or minivan" category?

Who owns the SUV and offroad category?

You see my point. There are a number of big segments that are not owned yet, and it is really unlikely that any one company can win them all.

If all social networks are converging on the same set of features, then eventually they will be close to equal in function. The only way to differentiate them will be in terms of the brands they build and the audience segments they focus on. These in turn will cause them to emphasize certain features more than others.

In the future the question for consumers will be "Which social network is most like me? Which social network is the place for me to base my online presence?"

Sue may connect to Bob who is in a different social network — his account is hosted in a different social network. Sue will not be a member of Bob’s service, and Bob will not be a member of Sue’s, yet they will be able to form a social relationship and communication channel. This is like email. I may use Outlook and you may use Gmail, but we can still send messages to each other.

Although all social networks will interoperate eventually, depending on each person’s unique identity they may choose to be based in — to live and surf in — a particular social network that expresses their identity, and caters to it. For example, I would probably want to be surfing in the luxury SUV of social networks at this point in my life, not in the luxury sedan, not the racecar, not in the family car, not the dune-buggy. Someone else might much prefer an open source, home-built social network account running on a server they host. It shouldn’t matter — we should still be able to connect, share stuff, get notified of each other’s posts, etc. It should feel like we are in a unified social networking fabric, even though our accounts live in different services with different brands, different interfaces, and different features.

I think this is where social networks are heading. If it’s true then there are still many big business opportunities in this space.

Our present day search engines are a poor match for the way that our brains actually think and search for answers. Our brains search associatively along networks of relationships. We search for things that are related to things we know, and things that are related to those things. Our brains not only search along these networks, they sense when networks intersect, and that is how we find things. I call this associative search, because we search along networks of associations between things.

Human memory — in other words, human search — is associative. It works by “homing in” on what we are looking for, rather than finding exact matches. Compare this to the the keyword search that is so popular on the Web today and there are obvious differences. Keyword searching provides a very weak form of “homing in” — by choosing our keywords carefully we can limit the set of things which match. But the problem is we can only find things which contain those literal keywords.

There is no actual use of associations in keyword search, it is just literal matching to keywords. Our brains on the other hand use a much more sophisticated form of “homing in” on answers. Instead of literal matches, our brains look for things things which are associatively connected to things we remember, in order to find what we are ultimately looking for.

For example, consider the case where you cannot remember someone’s name. How do you remember it? Usually we start by trying to remember various facts about that person. By doing this our brains then start networking from those facts to other facts and finally to other memories that they intersect.  Ultimately through this process of “free association” or “associative memory” we home in on things which eventually trigger a memory of the person’s name.

Both forms of search make use of the intersections of sets, but the associative search model is exponentially more powerful because for every additional search term in your query, an entire network of concepts, and relationships between them, is implied. One additional term can result in an entire network of related queries, and when you begin to intersect the different networks that result from multiple
terms in the query, you quickly home in on only those results that make sense. In keyword search on the other hand, each additional search term only provides a linear benefit — there is no exponential amplification using networks.

Keyword search is a very weak approximation of associative search because there really is no concept of a relationship at all. By entering keywords into a search engine like Google we are simulating an associative search, but without the real power of actual relationships between things to help us. Google does not know how various concepts are related and it doesn’t take that into account when helping us find things. Instead, Google just looks for documents that contain exact matches to the terms we are looking for and weights them statistically. It makes some use of relationships between Web pages to rank the results, but it does not actually search along relationships to find new results.

Basically the problem today is that Google does not work the way our brains think. This difference creates an inefficiency for searchers: We have to do the work of translating our associative way of thinking into “keywordese” that is likely to return results we want. Often this requires a bit of trial and error and reiteration of our searches before we get result sets that match our needs.

A recently proposed solution to the problem of “keywordese” is natural language search (or NLP search), such as what is being proposed by companies like Powerset and Hakia. Natural language search engines are slightly closer to the way we actually think because they at least attempt to understand ordinary language instead of requiring keywords. You can ask a question and get answers to that question that make sense.

Natural language search engines are able to understand the language of a query and the language in the result documents in order to make a better match between the question and potential answers. But this is still not true associative search. Although these systems bear a closer resemblance to the way we think, they still do not actually leverage the power of networks — they are still not as powerful as associative search.

A natural language search can understand the meaning of a query like “books about Harry Potter” and it knows this is not the same as “Books by Harry Potter.” But ultimately what is happening is that a linguistic expression is being converted into a more sophisticated keyword search. The language in the query is being mapped to documents that contain text that answers a question, or to data objects that match the thing being asked for. This is certainly better than keyword search but it is still ultimately just a smarter form of literal matching. It is not really making use of associative search along networks of semantic relationships in the data (other than linguistic relationships between words in the query) or any sort of sophisticated reasoning.

By comparison, associative search doesn’t merely understand the meaning of the query, it understands and can reason about relationships in the data. This is an important distinction.

An associative search returns documents that represent things that are related, via various forms of associations (semantic links), to the things in the query. An associative search looks through a network of associations for the things that are most connected to the items in the query. By specifying more specific starting points, the set of things which are connected to all those starting points is narrowed. Thus an associative search is an intersection of multiple networks. The items that are most strongly intersected are the results that are most likely
to matter.

Associative search is a very different approach to search from keyword search (which merely looks for things with the keywords in them) and natural language search (which merely looks for things that contain content that matches the meaning of the question). It also happens to be more similar to how our brains actually think.

On its own, associative search represents an important advance in the way we search. But by adding some simple reasoning to an associative search it becomes even more powerful. Reasoning adds the ability to generalize or get more specific, and to weight various paths through the network of relationships in more sophisticated ways, such as based on logical relationships or inferences through the network.

A simple example of reasoning is transitivity — for example, if A is a part of B and B is a part of C, then A is a part of C. If we know that the “part of” relationship is transitive, then whenever we see chains of “part of” links between things we can make transitive inferences. In an associative search these inferences are quite useful. For example, we can search for all the parts of a 747 jet. Using transitive reasoning along networks of relationships we can find all the parts, even those things that are “parts of parts.” Similarly we could find “all products of Sony” including products of subsidiaries and business units of Sony. Transitive inferences across transitive links is just one type of reasoning; there are many other variations
that are possible, which when combined together become even more useful.

Our current search tools — whether they are keyword based or natural language based do not support true associative search, let alone reasoning. But we do see associative search starting to appear in a very different breed of application: social networks. A search in LinkedIn for example, is an associative search. Will social networks do an end-run around traditional search engines to provide the next-generation of search? It’s quite possible. Facebook and LinkedIn are far better positioned than Google today for associative search. In fact, I would venture that this is how Facebook could give Google some serious competition. But they have to hurry if they are going to do this — Google has clearly realized the power of “social search” and is rapidly moving to leverage it in their own search results.

Ultimately associative search is more than just social search however. To be really effective, associative search engines need to understand and leverage the full spectrum of relationships between things, not just social relationships. They need to see and understand more types of relationships between more types of things. In order to accomplish this, associative search engines need the Semantic Web.

The Semantic Web provides exactly what is needed to enableassociative search, with reasoning, on the Web-at-large. Using RDF and OWL, content can be marked up with metadata that specifies not only its intended meaning and structure, but also the various kinds of semantic relationships it has to other content and to other concepts. In other words, these standards provide a way to add a new network of
semantically defined associations to the data on the Web. For example a document about Microsoft can be linked to the concepts “Software Company,” “Software Manufacturer,” and “Redmond.” It can also be linked to a data record that represents “Microsoft” and the properties that define it as a company. The “Microsoft” object can then link to companies that are “suppliers” and “customers” and “competitors” as well as to things which are connected as “products” or “services.”

This rich network of relationships between things goes far beyond documents. It contains relationships to people, places, other organizations, products, events, services, etc. It’s similar to a social network, but instead of just containing people and social relationships, it contains more types of things and relationships between them. This is really what the Semantic Web enables. One can imagine that as this new semantic data becomes visible on the Web (which is rapidly happening in fact), the power of search will be dramatically improved. Associative search is coming soon to a Web near you!

With that in mind, here is an example of how Semantic Web enabled associative search will work in the future.

PROBLEM: I am trying to remember name of the organizer of a conference I once attended.

WHAT I ALREADY  KNOW:

• I know this person and have corresponded with them in the past.
• The conference was related to government and the Internet.
• It took place in a town near Big Sur, but I can’t remember the name of the town.
• The organizer of the conference once introduced me to a male celebrity, but I can’t remember the celebrity’s name.
• I gave a talk at the Conference about Web 3.0.
• My friend, Sue Smith, also spoke at the conference.
• The conference I attended took place in the Spring, but I am not sure if it was last year or two years ago.

In the above example, I cannot remember the specific keywords that will help me generate a query to find the answer. Instead, I remember a number of relationships and generalizations about the answer. Present day search engines cannot see these relationships, and they have no ability to understand a generalization and look at things it contains.

The ability to intersect the sets formed by relationships and generalizations is a fundamental feature of human memory and search. But our present day tools don’t have these capabilities. Thus we have to spend time translating our questions into keywordese, rather than just asking our questions in the actual language of human thought.

There are two ways to approach solving this.

1. The first way is to create artificial intelligence which, given a question in natural language English, can understand it and reason about the question as well as understand and reason about the information in the set of documents being searched, in order to intelligently arrive at candidate answers. This is computationally intensive, and very hard to program. This is why AI hasn’t quite happened yet on this scale.
2. A perhaps easier approach is to use the Semantic Web. In the Semantic Web approach, metadata is embedded into content that describes the meaning of the content, it’s various important properties, and its relationships to other concepts. On the basis of this metadata, the problem becomes much simpler to solve. Instead of doing high-level AI it becomes essentially a statistical search.

Now let’s look at how using the Semantic Web could help us solve the above problem via an associative search:

Items are connected to more general or specific concepts by virtue of semantic linkages between concepts. For example, the conference I am looking for is related to the concepts “Government” and “Technology.” If I can at least remember that then I can find conferences related to government and technology. Furthermore, since the concept “Policy” is a subset of government it may be related to that topic
as well.

Likewise, things are connected to things that are “near” them via geographic links. Because the conference was near Big Sur it is in Northern California, along the coast. It is probably in a town that is geographically close, ror example Carmel-by-the-Sea is a town that is near the Big Sur area.

The organizer of the conference introduced me to a male celebrity. There are several celebrities in my social network. If the fact that I met certain people via introductions from other people was stored using semantic links, then this too would be searchable. For example, “find all celebrities I was introduced to by my connections” would be a solvable query. Similarly, “find people who introduced me to celebrities” would also be solvable.

The fact that I gave a talk at the conference could also be semantically represented on a data record describing the conference, as well as on my own profile. Thus there could exist a link such as “speaker at” which links me to various conferences I have spoken at. I could then get a list of all the conference I have spoken at. I could also look for all the conferences where both myself and Sue Smith were speakers.

Or, better yet, there could be a link called “Gave talk about” which links me to an instance describing each talk I have given. From such an instance there could then be “Gave talk at” links to all the events where I have given that talk. So I could look up my “Web 3.0″ talk and then see all the conferences where I gave that talk.

Temporal relations can also be generalized and semantically represented. For example, the conference I am looking for took place in the spring. Therefore only look for conferences that took place in or near months that are considered to be in the spring season.

By intersecting the results of the above searches we narrow down very precisely to a set of people I might be looking for, or just to a single qualifying person.

For example the answer I was seeking for was that the organizer was named Robert Jones, and the conference was about Government and Technology Policy in Carmel-by-the-Sea last spring. This result should be easily findable via associative search starting from the above set of things I remember.

But if for some reason the answer is still not there, there is another capability which the brain uses that we need to add to our search engines: Perturbation, or what could be called “prospecting.”

The query I entered is comprised of a question and a set of facts related to the answer I am seeking. But there is a possibility that I asked the question incorrectly, or some of the facts I added were incorrect, or insufficient. Perturbation can correct for this by introducing variations into the question and the facts in order to explore the space of answers that are “near” them as well.

There are many ways to go about adding perturbation to the system — for example, we can search more than one hop out from every link, or we can search for other types of relationships that are highly correlated with relationships we are asking for
explicitly, or we can include results for things which are strongly connected to things that are found.

From a user-interface standpoint perturbation can be controlled with a simple “sliding lever” in the user interface for “Precision.” If the user sets very high Precision as a requirement then there is no perturbation — the results are exact matches to the query and facts. If there is low Precision as a requirement then there can be more perturbation, thus the results are fuzzy and may include things that are near what I asked for but not exactly what I specified, enabling me to discover
things via relevant relationships that I could not even remember to mention as facts.

Finally, using a reasoner, the results found by the above search can be analyzed such that those results which are most likely to be what I am looking for, given the facts I have included as constraints, are presented first. Reasoning becomes the ranking algorithm in the system, rather than something like Pagerank. The answers that actually make the most sense in the context of my question are delivered first.

The above illustration describes how searches that are powered by the Semantic Web will work, once this technology is widely adopted. This is how the brain works, and how our search engines should work as well.

This is not a pipedream — in fact it is already happening in research settings and in the government. Within 15 years, if not a lot sooner, we will see these capabilities emerge in consumer-grade search interfaces.

This is a five minute video in which I was asked to make some predictions for the next decade about the Semantic Web, search and artificial intelligence. It was done at the NextWeb conference and was a fun interview.

Learning from the Future with Nova Spivack from Maarten on Vimeo.

Carla Thompson, an analyst for Guidewire Group, has written what I think is a very insightful article about her experience participating in the early-access wave of the Twine beta.

We are now starting to let the press in and next week we will begin to let waves of people in from our over 30,000 user wait list. We will be letting people into the beta in waves every week going forward.

As Carla notes, Twine is a work in progress and we are mainly focused on learning from our users now. We have lots more to do, but we’re very excited about the direction Twine is headed in, and it’s really great to see Twine getting so much active use.

Vewr

I’m here at the BlogTalk conference in Cork, Ireland with a range of bloggers and technologists discussing the emerging social Web. Including myself, Ian Davis and Paul Miller from Talis, there are also a bunch of other Semantic Web folks including Dan Brickley, and a group from DERI Galway.

Over dinner a few of us were discussing the terms “Semantic Web” versus “Web 3.0″ and we all felt a better term was needed. After some thinking, Ian Davis suggested “Web 3G.” I like this term better than Web 3.0 because it loses the “version number” aspect that so many objected to. It has a familiar ring to it as well, reminding me of the 3G wireless phone initiative. It also suggests Tim Berners-Lee’s “Giant Global Graph” or GGG — a synonym for the Semantic Web. Ian stayed up late and put together a nice blog post about the term, echoing many of my own sentiments about how this term should apply to a decade (the third decade of the Web), rather than to a particular technology.

The Crunchies are done. At Radar Networks we are really honored to have our product, Twine.com, nominated as a finalist for Best Technology Innovation of 2007. It was very cool to see our Twine logo up there on stage next to Facebook, Digg, LinkedIn and so many other incredible companies — especially considering we were the only company that was still in closed Beta in the awards (and yes, we are coming out of closed beta in March, so get ready!).

Meanwhile, one of things that made the Crunchies fun was that every company was asked to submit a video. Not all companies did, and not all of them were that creative. Some however were really funny, including ours. Here is a link to the "director’s cut" of the Twine Crunchies video for 2007. Enjoy!!!

ps. For those who don’t live in the USA… CoolWhip is a synthetic dessert topping we have here in the States. Imagine whipped cream, made out of some kind of industrial byproduct. It actually tastes pretty good, whatever it is. And it has almost no calories — possibly because there is nothing in that is actually digestible by humans. It’s really a wonderful technological innovation. Thus our choice.

My company’s product, Twine.com, has made it to the finalist round in the Crunchies, a new annual tech industry awards competition, under the Best Technical Achievement category. Please help us win by casting your vote for Twine here. Thanks!

UPDATE: It turns out, that for some odd reason the Crunchies allows each voter to vote once per day per category — in other words, you can vote multiple times in the same category — one vote per user per day — so please vote for Twine again if you can.

Scoble came over and filmed a full conversation and video demo of Twine. You can watch the long version (1 hour) or the short version (10 mins) on his site. Here’s the link.

Last month I was on a panel about Semantic Web Opportunities at the MIT / Stanford Venture Lab, at Stanford University. The panel was moderated by Paul Saffo, and included myself, Robert Cook, Alex Iskold and Paul Kedrosky. The full video of the panel is online. You have to register to view it, but registration is free. Here’s the link. I should also note this panel is for a business school audience that doesn’t know much about the Semantic Web or the related technologies, but it’s fun, full of laughs, and an interesting conversation. Worth watching!