Today I’m happy to announce the launch of StreamGlider, a new tablet app (initially on iPad) that provides the first live streaming dashboard for keeping up with your interests.

TechCrunch just broke the story.

The inspiration for StreamGlider was a product that launched in the early 1990′s called Pointcast. Pointcast streamed news, entertainment, ads and other updates to screensavers. Pointcast was great, and we, (myself and my co-founders, Bill McDaniel and John Breslin) wondered whether we could evolve that concept and update it for the tablet and mobile era.

We designed StreamGlider to be the ultimate live streaming newsreader. It does what you have come to expect, plus a lot more. And it does it live – it streams live updates to your tablet.

It also offers a lot of new functionality that supports new ways of using a reader.

• StreamGlider pulls live updates from content sources on the Web (RSS feeds, Google Reader, and Web API’s like Twitter, Facebook, Youtube, Flickr, etc.) onto mobile devices, and displays them in a variety of formats.
• It can function as a live digital picture frame for the Web, showing news articles, photos from friends, videos, etc. as full-screen slides that scroll past.
• It can also show streams as a live interactive filmstrips that function like tickers.
• And it can show streams in an interactive magazine format that is similar to a newspaper layout.
• You can also play and watch videos in StreamGlider.

Powerful Features

StreamGlider is fully gesture controlled – everything can be controlled by swiping, pinching, pointing, tapping, etc. You can easily customize the streams you want.

You can also create mashups of streams that pull from many different sources on a theme – for example you can pull from different news sources about sports, or different photo and video sources about a topic.

In addition to all this, you can make very personalized streams that pull from your social media accounts, and filtered streams that search for particular topics in content sources.

StreamGlider is also social. You can share individual items, or even entire streams of items, with your friends.

We designed StreamGlider to be brandable. Partners and customers can create their own private-labelled versions of StreamGlider, with their brand and their content, for their audiences. Brands can sell it or give it away free and run ads in it if they want. (Contact StreamGlider, if you’re interested in doing this for your brand).

This frees publishers, brands, and enterprises to create their own powerful readers for their audiences, with their brand, instead of having to live inside of other apps like FlipBoard or Pulse. They can have their own icon on the desktop and keep their direct relationship with their customers.

There are many use-cases for this – for example, you might want to distribute your own branded StreamGlider for your publication, or for a consumer product, or to your fans, or for a big event, or to your customers or employees. There are many reasons to do this – and you don’t have to be a software company to do it – you can almost instantly get your own branded StreamGlider.

We also designed StreamGlider to be open-source in the future. More news on that later. We hope we can become the Mozilla of newsreaders.

What’s next?

The team behind StreamGlider has a long history of making smart, semantic apps. You can expect that in future versions of StreamGlider, the app will begin to get smarter, more personalized, and even more social. This is just the beginning of our roadmap.

We will also be adding in support for more types of streams. Stay tuned!

Meanwhile, you should check it out. Download it to your iPad and see what I’m talking about.

After my former project, Twine.com, was sold, I began to turn my attention to the Next Big Challenge: How to make sense of the growing real-time Web, or what many call, “the Stream.”

I could see the writing on the wall, and it was less than 140 characters: Social media’s own success was going to be its biggest challenge. The Stream was going to soon become unusable.

In the early days of the Stream, it was actually possible to keep up with your community on Twitter and Facebook effectively. Not anymore. There are just too many people messaging too often. The chances of even seeing a message before it scrolls into history are getting lower every day.

Today, the Stream is growing exponentially. Twitter famously grew by 3x in the last year and sends out more than 250 million Tweets per day. Facebook sends billions of public and private messages per day. And this is just the tip of the iceberg — or the deluge, as it were.

There are so many new and growing sources of messages in the Stream: Google+, LinkedIn, Foursquare, Youtube, RSS feeds, and more are coming. And that’s just the consumer side of the Stream – there’s a whole other side to the Stream: Chatter, Yammer, Socialcast, Jive, and many other enterprise streams are also growing rapidly.

And on top of this there is a whole new deluge of machine and app-generated data that is just starting to join the stream, and may eventually dwarf human-generated data.

At the same time as all these new networks are popping up to enable messaging in the Stream, the barrier to creating and sharing messages has also never been lower. I call this The Sharepocalypse.

It’s never been easier to share — People are sharing more kinds of information, more often, with more people, than ever before. And it’s requiring less thought too — because the messages themselves are so short. This is resulting in a collective overshare of unimagined proportions.

With email, the messages were usually long and required some effort, so people sent relatively few emails per day. And at least with email there were some basic social rules about what you could send to everyone without being a spammer.

Not anymore. In the age of the Stream it’s quite normal to post out what you had for lunch, or some cool product you are looking at in a store window, with a photo, to the entire world. That would have been unthinkable in the email era. In the age of the Stream, it’s not even an afterthought. The Sharepocalypse is here, in spades.

The result of all this adoption and growth of the Stream is a new kind of information overload, stream overload.

Stream Overload is worse than email overload, because it includes email overload.

Email, in my opinion was “Stream 1.0.” Social media (RSS, Twitter, Facebook, etc.) were “Stream 2.0.” And now we’re entering “Stream 3.0″ – when everything – all information, all applications, everyone, even things – become part of the Stream.

(Yes I know, version numbers are so Web 3.0, but it’s helpful to use them as handles for the discussion. Stream 3.0 is indeed a different era from the early days of the Stream.)

We’re already seeing the signs of stream overload — but this is just a preview of what’s to come as Stream 3.0 comes to maturity. The growth of the Stream is still only just beginning. Most of the planet isn’t using it yet. And most people don’t realize how integral it’s going to be in their lives in coming years.

If the Web is the planet’s brain, the Stream is its mind – it’s the living, breathing, thinking, learning, aware, acting part. And we’re all going to be part of it 24/7, whether we like it or not. So it better be good, it better be smart, it better be useable, or we’re all going to be gridlocked and buried in messages we don’t want.

And this is the Next Big Problem: The Stream is going to become both more important, and more noisy at the same time. This is a classic crisis. Either something must be done to reduce the noise, or it’s not going to be useable. And this will lead to problems, because it’s important that it actually is usable.

What happens if the Stream really breaks down under its own weight?

If the signal-to-noise problem isn’t solved, and people can’t keep up with the Stream, they’re going to give up. They’re going to stop paying attention. They’re going to stop trying to keep up. They will never be able to scroll down enough. They won’t even login to sites like Twitter and Facebook if they are too overloaded.

And if nobody is there listening, then there won’t be much point in posting news and updates to the Stream either. People will stop posting too.

And without the people there, marketers won’t post either – so the advertising money will go away. And even in the social enterprise, if streams for teams get too noisy, they will also stop being used and people will move to some new solution.

And without the people there, the Stream will become an automaton. All that will be left is machines posting to machines.

Unless something is done to solve it, of course.

And something IS being done, it turns out. We’re launching Bottlenose tonight. To read more about the history of the project, read Bottlenose has Launched!

Notes

Make sure to follow us on Twitter:

• @bottlenoseapp – the official Bottlenose Twitter account
• @novaspivack — yours truly
• @dominiek — Dominiek ter Heide, Bottlenose CTO

And come check out Bottlenose! The app is still in invite beta so you either have to have a high enough Klout score or an invite code to get in.

The first 500 readers of my blog who want to try it out, can get into Bottlenose using the invite code: novafriends

Check out the what the press is saying about Bottlenose:

Bottlenose Intelligent Social Dashboard Launches Private Beta  — ReadWriteWeb

Bottlenose is a Game Changer for Social Media Consumption — Mashable

Bottlenose is a Social Media Dashboard That Makes Sense of the Stream – Venturebeat

Can This Startup Eliminate Social Media Overload? — Inc.

The Day of the Dolphin: Swim in the Personalized Stream With Bottlenose — SemanticWeb

Bottlenose Launch – A Smarter Way to Skim the Stream – SiliconAngle

Bottlenose is a Web-Based Twitter Client for Power Users — AllThingsD

Managing the Sharepocalypse — AdWeek

Can Bottlenose Help Prevent the Social Sharepocalypse? – GigaOm

Social Overload? Bottlenose Promises Intelligent Filtering — Information Week

Today, after almost two years of work in stealth, I am proud to announce the launch of Bottlenose.

While I have co-founded and serve on the boards of several other ventures (The Daily Dot, Live Matrix, StreamGlider, and others), Bottlenose is different from all my other projects in that I am also in a full-time day-to-day role as the CEO. In short, Bottlenose is what I’m putting the bulk of my time into going forward, although I will continue to angel invest and advise other startups.

The story of Bottlenose began when my good friend and advisor, Josh Jones-Dilworth, introduced me to Dominiek ter Heide after I sold my last company, Twine.com in 2010.

Dominiek was at the time working on a new kind of personalization technology for social media. Meanwhile, I had been thinking about how to filter the Stream, and the emerging problem of the Sharepocalypse and what I have been calling “the Stream 3.0 Problem.”

Josh knew both of us and had a hunch that we were really thinking about the same problem from different angles. Dominiek and I started speaking via Skype and soon we teamed up. Bottlenose was officially born in 2010.

Working with Dominiek has been a true pleasure. He’s one of the most productive, talented, software engineers I’ve ever met. It’s been an amazing ride so far. Soon, thanks to Dominiek, we were joined by an A-team of killer engineers with expertise in natural language processing, Node.js, Javascript, HTML 5, machine learning, cloud computing, NoSQL, and more.

Our little band of hotshots has produced an amazingly robust and powerful app — something that even large companies with huge engineering teams would be hard-pressed to develop. I’m honored to be working with these guys, and very proud of the team and the what we’ve built.

We have also been fortunate to be joined by some terrific angel investors, including Andy Jenks, of Stage One Capital, and several others (see the About page on Bottlenose for the complete list).

So what is Bottlenose anyway? Well one way to find out is to visit the site and check out the Tour there. But I’ll summarize here as well:

Bottlenose is the smartest social media dashboard ever built. It’s designed for busy people who make heavy use of social media: prosumers, influencers, professionals.

Bottlenose uses next-generation “stream intelligence” technology to understand the messages that are flowing through Twitter, Facebook and other social networks. It also learns about your interests.

On the basis of this knowledge, Bottlenose helps you filter your streams to find what matters to you, what’s relevant, and what’s most important. Bottlenose also includes many new features, like Sonar, which visualizes what’s going on in any stream, and powerful rules and automation capabilities to help you become more productive.

This is just the beginning of this adventure. Our roadmap for Bottlenose is very ambitious, and it’s going to be a lot of fun, and hopefully will really make a difference too. We’re super excited about this product and we hope you will be as well.

Check back here for more posts and observations about Bottlenose and where I think social media is headed.

Make sure to follow us on Twitter:

• @bottlenoseapp – the official Bottlenose Twitter account
• @novaspivack — yours truly
• @dominiek — Dominiek ter Heide, Bottlenose CTO

And come check out Bottlenose! The app is still in invite beta so you either have to have a high enough Klout score or an invite code to get in.

The first 500 readers of my blog who want to try it out, can get into Bottlenose using the invite code: novafriends

I look forward to seeing you Bottlenose!

For more about the thinking behind Bottlenose, read The Problem of Stream 3.0

Today I’m pleased to announce that, The Daily Dot, our newest “venture production,” has launched into public beta.

The Daily Dot is the first of its kind – it’s the Web’s newspaper — the first community newspaper about the Web. We cover the Web like a town paper covers its community. Here’s a video overview of the site.

This venture began with the insight that each of us is spending an increasing amount of our lives online, in various online communities, yet we have very little insight into what’s going in this new landscape. These communities are literally places, and some of them are quite large. This is beautifully illustrated in this “map” of the Web as a geography.

I believe that it’s time for the Web community to have it’s own newspaper. The launch of the Daily Dot — the web community’s first actual newspaper of record — is a turning point, a coming-of-age, for the Web as a medium, as a place, and as a community.

Our editorial focus is different than other publications that cover the Web. Instead of covering the Web as an industry, a technology or a phenomenon, we cover it as a community. We tell the stories of the people, culture, content, events and issues that are making waves in communities around the Web. And to find and report on these stories, we have embedded reporters in those communities: Facebook, YouTube, Reddit, Twitter, Tumblr, with more communities coming soon.

Just like our physical cities and towns, our online communities are constantly moving and developing, and they are full of interesting people doing newsworthy and important things. The Daily Dot’s mission is to cover these communities just like physical community newspapers cover cities and towns.

Where a town newspaper covers the latest high school sports game, the town meeting, the local crime report, we cover the story behind the hottest viral video sweeping the planet, the latest social movement in Facebook, and important issues (like cybercrime or online bullying) that are happening in our online neighborhoods.

When a major event happens in the physical world – like the revolutions in Arab world, for example — we don’t cover the events themselves, we cover their online footprint — what’s happening online that relates to the story.

The Daily Dot will also cover what’s happening around the Web in time: just like physical community newspapers have calendar sections – The Daily Dot has an online events section, provided in partnership with Live Matrix, one of our other venture productions, that aggregates the schedule of the Web. These two companies are highly synergistic and form the beginnings of our online media network.

While those of us in the Web industry have our fingers slightly more on the pulse of the Web, the vast majority of people who use the Web do not read industry blogs and have little or no visibility into what’s going on in the online world or where it’s headed. Other than a few articles a week published by mainstream media, they are not being informed.

It’s time for that to change. The Daily Dot will be publishing dozens of articles each day about what’s happening online. We’re writing for the mainstream, not for elites or geeks. The Daily Dot is for the people who use the Web — who live in it — not just the people who are building it.

Our content is designed to be entertaining, interesting, informative — and sometimes edgy and controversial – kind of like People Magazine meets USA Today, with a little bit of TMZ thrown in.

If you want to know what’s happening online, or you’re looking to find the hottest emerging entertainment, personalities, viral videos, issues, etc — and the stories behind them — The Daily Dot is your newspaper.

But The Daily Dot is not just a newspaper, it’s also a very interesting business venture. It’s a chance to build what could become one of the largest circulation newspapers in the world someday – a global newspaper about the one community that we all share in common, no matter where we actually live.

I also want to congratulate and thank the amazing editorial and development team at the Daily Dot, who made this possible. And most importantly, I want to acknowledge Nicholas White (Daily Dot CEO), Owen Thomas (Daily Dot founding editor), and Josh Jones-Dilworth (marketing guru), my co-founders in this venture.

Nick and Owen are leading business and editorial, and running the operations, and Josh and myself are on the board, advising to help in our respective areas of expertise. Nick and Owen deserve all the credit here — they have done the heavy lifting to bring this vision to market, and I’m very proud to be working with them.

Please join me us helping to spread the word about The Daily Dot — it’s your newspaper — and we need your help to make it great (and we look forward to your feedback and participation in the comments).

This is going to be a fun ride and I can’t wait to see how it evolves.

The social media landscape is changing quickly, but this change won’t be immediate, or for that matter, efficient. And that’s going to be a big problem for all of us.

I believe that Twitter, Facebook, Google+ and LinkedIn are fundamentally different, and thus, should not be in competition. However, I’m not sure the companies themselves see it this way. It’s likely they will continue dedicating resources to competition instead of differentiation.

And while the social media gods fight it out in the clouds above us, what will happen down here on Earth? What about all of us, the little people — the users?

We’re entering a new era of social network chaos, and this, in turn, is going to create new needs and opportunities for startups.

The Sharepocalypse

Welcome to the “Sharepocalypse,” a new era of social network insanity.

READ THE REST OF THE ARTICLE HERE

It may look like Google+ is competing with Facebook and Twitter, but I don’t think that is what will happen in the end. I think Google+ is a very different kind of service and it’s not clear that it can or will, or should, replace these other services.

In a series of articles here on my blog, I’ve explained the differences between these services, and what Google+ is really for and what it means for the rest of the social media giants:

1. Google+ is Really for Sharing Knowledge, Not Social Networking
2. Should Facebook be Worried About Google+?
3. Why Twitter’s API Strategy Must Change in a Google+ and Facebook World
4. Why the Google+ Developer Ecosystem Will be Different from Twitter

The conclusion I draw from all this is that instead of one social network to rule them all, I think it’s more likely that the social media landscape is going to divide into different territories, with each of the major social networks playing a different role.

Here’s how I think this all going to shake out:

• Facebook is for social networking
• LinkedIn is for business networking
• Google+ is for knowledge networking
• Twitter is for notifications

They just don’t know it yet.

Here is some more detail on this idea:

• Facebook is for social networking
  • Facebook is the new social infrastructure for the planet, and Google+ is no match for it. By social, I mean non-professional, personal, friend-to-friend and group communication. There’s a lot more happening in Facebook than this however: gaming, branding, groups, marketing. But all this other activity depends on the fact that people spend so much time in Facebook, socializing. This is very different from what’s happening on Google+ and Twitter as well.
• LinkedIn is for business networking
  • It’s the infrastructure for professional networking in the old-school sense – as in getting a job, finding customers, locating partners, hiring people, doing biz dev and sales, etc. LinkedIn is the most differentiated and focused of all these players: they know what they’re good at and they’re not trying to be all things to all people. Now LinkedIn needs to build more bridges into more third-party applications and services to keep people aware of it and using it.
• Google+ is for knowledge networking
  • Google+ is an infrastructure for sharing knowledge, not social networking. Knowledge has always been Google’s strength and core focus. Knowledge is not just articles, but the conversations around them, and these conversations are one of Google+’s best features. More importantly, because Google has such a powerful search infrastructure, and such a powerful computing architecture, they are in a position to combine Google+ with search and massive analytics and machine learning, to dynamically re-organize and connect both the Web and the real-time Stream. By doing this Google+ could be a potential successor for the Blogosphere, and could leap far ahead of other competing search engines as well.
• Twitter is for notifications
  • Twitter is really a notifications infrastructure. That’s what they do best, and what they should be focusing on. They are executing on the wrong strategy right now. They are trying to be a media company, but that is not their strength and others already are far ahead of them at that. But as an infrastructure for short notifications, Twitter has an opportunity to be unique and win, if they focus on that. Twitter has replaced RSS, for better or for worse, as the primary way people and applications share and track these kinds of notifications. Twitter could leverage this position to become the notifications infrastructure for the whole world – and for all of the other networks – even for G+ and Facebook – if they played their cards right and stopped focusing on competing for eyeballs.

In previous articles, I’ve written about how Google+ can build a developer ecosystem on Chrome that is different from Twitter’s ecosystem, and how Twitter must change to survive against that. It’s clear that Google+ and Twitter are very different animals.

Now what about Facebook? Should Facebook be worried about Google+? Are Facebook and Google+ really competitors? I don’t think so.

Google+ is not as geeky as Twitter, but it’s still too complicated for most consumers to want to use it.

Figuring out how to use Google+, and how to make effective use of it, at this early stage, is like trying to use an old shortwave radio. Actually, it’s like trying to figure out a shortwave radio that is only halfway built. This is not an activity my mom is going to enjoy.

It’s going to be a while before Google+ is ready for primetime consumer use. Facebook is way ahead on that front.

And there’s also the fun factor issue — Facebook has focused on fun: games, pokes, virtual gifts, and all sorts of social silliness that consumers just love.

The lack of play in the Google+ experience is actually a plus, not a minus, for many early users – there’s more signal, less noise, there – at least potentially. And this creates a self-selecting use-case: people are using Google+ for sharing ideas and having real conversations (and as of week two, not only about Google+ it turns out).

As of this article there is certainly an increase in non-serious content showing up on Google+, but it’s still a drop in the bucket compared to Facebook’s content mix. This could be an early-adopter effect that could change if more mainstream users adopt G+, but currently, my instincts are telling me G+ content is going to be more serious than fun. I’m not convinced the mainstream consumer audience is going to use G+ for fun.

Google+ is best used for sharing knowledge. This may result in Google+ filling a role that USENET used to play and that the fragmented blogosphere never really succeeded at solving: a unified knowledge sharing and conversation medium.

Hopefully the folks at Google+ will realize that the slightly more serious communication that’s happening in the service is a good thing. Instead of trying to change that by introducing more ways to play, they might want to consider celebrating it.

Keep out the silly social games, don’t introduce the fluff. This will preserve Google+ as a higher signal-to-noise communication channel and will make it unique from Facebook.

Hopefully Google+ won’t immediately integrate Zynga, for example, because that would totally ruin their differentiation from Facebook and take them in a direction they have no in-house DNA for: fun and games.

It’s just not too likely that the serious engineering and science culture of Google can replicate the lightheartedness of Facebook. And anyway if they could make Google+ fun, will anyone want it? After all they already have Facebook for that.

People are not going to use Facebook for serious conversations – it’s already too late for that. And they’re not going to use Google+ for superpoking. They can already poke each other to death perfectly well in Facebook.

Google+ is different from Facebook. And that’s a good thing for both companies. There may actually be room for both of them in this town.

As a result of the emergence of Google+, Twitter could soon find itself in a tough spot. A large chunk of their core developer base might migrate to Google+ because there is simply more opportunity there.

Why? Well for starters, it’s really easy to crank out Chrome extensions and you can market and sell them instantly in the Chrome Web Store to a ginormous captive audience that is many multiples of the size of Twitter’s user-base. I’ve written about how Google+ can leverage Chrome to build an ecosystem here.

And if you succeed, your shiny new Google+ feature might even get you bought by Google for a million bucks. What engineer wouldn’t want to spend a few weeks making a feature that could net them a million bucks and a job at Google in a few months?

Compare that to what it’s like to be a Twitter developer today. Twitter has no plugin framework, no app store, no browser, no OS, and they are clamping down on their API terms of use, and even actively going to war against some of their third-party developers. And they don’t have the kind of acquisition budget or appetite that Google has. Twitter has only made a few acquisitions to date.

To make matters worse for Twitter, there’s very little loyalty to Twitter among Twitter developers right now – mostly there’s fear because of the recent Ubermedia and Tweetdeck situation, and Twitter’s recent moves to add their own photo-sharing, and soon their own analytics.

What opportunities are there really for developers on the Twitter platform, that Twitter doesn’t actually want for itself? Twitter has suggested that it wants it’s developers to “move up the value chain,” but to what exactly? How high should they jump? And if they do, will Twitter just pull the rug out from under them when they land?

This kind of FUD may very likely drive Twitter’s core third-party app developers over to the seemingly greener and safer pastures at Google+. And on Google+ developers can rapidly crank out new features as Chrome extensions, they don’t have to use an API. And this gives them instant marketing to a huge captive market of Chrome users too.

Now it’s worth noting that being a Google+ Chrome extension developer won’t necessarily be safer than developing on the Twitter API in the long run. But it will seem safer for a while, and that will be enough for many developers to go there.

Like Twitter, Google will be able to cherry pick the best opportunities on its platform. Any Chrome extension that really becomes a big hit on top of Google+ will be either acquired or copied by Google, and since Google owns the means of distribution (Chrome and Google+) there will be no competition for such deals (what buyer would compete with Google to buy a Chrome extension that Google wanted to own?).  But there is at least a 12 to 24 month window for developers to create value and potentially get bought by Google before Google starts competing with them.

Meanwhile, while their developers start moving to Google+, Twitter is likely to continue to focus on being a media company. This could be a fatal mistake.

Twitter simply does not have the reach of Google. They will never have it. Google is simply everywhere. It’s a completely hopeless battle to try to be a bigger destination than Google. Google has already won that battle. Twitter will never be as big as Google.

What Twitter DOES have — which Google does NOT have (yet) — is a massive installed base of third party apps publishing and subscribing to their message stream API. Assuming Google+ doesn’t come out with an API quickly, and that they drive innovation onto Chrome before they release a full API, there is a window of opportunity for Twitter to beat Google on the API front.

If Twitter focused on building around their real strength, their API, as I have suggested previously, instead of trying to become a media company and destination, they could have shot at long-term prosperity and differentiation as the messaging infrastructure of the planet. That’s a much bigger play for Twitter than being a media company, and it’s something Google+ is not positioned for. Twitter could win this.

(So why aren’t they doing this? What is Twitter’s management thinking? If you think you know, please comment on this article with your theory)

Twitter does not have the distribution and platform leverage that Google has, nor the huge installed base that Facebook has. And they have another problem: Twitter is still too geeky for mainstream consumers.

It’s just too hard to learn to use Twitter’s syntax properly. And the 140-character limitation results in all kinds of geeky abbreviations and conventions in the content and social behaviors in the system. Compared to other apps like Google+ and Facebook, which support long messages, richer text, and real threaded discussions, Twitter is going to seem cryptic and retro – like IRC.

No offense to Twitter – They’ve done something amazing. And I love geeks and count myself as one of them. So I totally get and like the geekiness. But it’s not going to work for mainstream consumers in the long-term.

Unfortunately, geekiness is hard-wired into Twitter’s DNA. It’s in the syntax of the app, their user-experience, and their culture. It’s also the in DNA of the core of their audience. So it’s not something that’s going to be easy to change. But to win the eyeballs war – the consumer war – you just can’t be that geeky.

So either Twitter has to undergo gene therapy to completely change their DNA to become a lot less geeky (unlikely), or they need to embrace their inner-geekiness and focus on their API and developers again: Cater to the geeks. Love the geeks. Make other geeks rich.

Is the Twitter/Apple deal the solution? Perhaps Apple could eventually buy Twitter and perform gene-therapy on them, transforming them into a more consumer-friendly product company. But if that doesn’t happen (and I doubt it will) then a deal with Apple is probably not enough to transform Twitter into a mainstream consumer product.

The key is that Twitter is not the same kind of animal as Facebook or Google+. Twitter is not a media company; it’s a notification company. It’s insufficient for creating rich content, or building rich conversations, but it’s great for short one-off notifications – and the 140-character limit is actually a good thing when viewed from this perspective.

Instead of trying to be a media company, Twitter should pivot back to fundamentals and focus more on their notifications API. This is what they do best. They should do this soon, and while they are at it, they should encourage third-party clients to build on this API again, instead of discouraging them.

By doing this right Twitter could become be the publish-subscribe messaging architecture for the world – including even for the other messaging networks like Facebook and Google+.

That’s seriously frikkin huge. And it’s unique too: It’s not something that Facebook or Google+ are technically designed or positioned to do. It’s what Twitter does better than anyone else, and it’s really what everyone is using Twitter for anyway.

How Twitter Can Win As an API

As an API focused company, Twitter could be woven into literally every app and service in the world as the means of publishing and subscribing to notifications of all kinds: Notifications between people and people, notifications between people and apps, and even notifications between apps and apps.

If they did this right, people might even use Twitter to keep up with notifications from Facebook, LinkedIn and Google+, as well as every publisher, other apps, and individuals.

Twitter wouldn’t necessarily be where the content is created or where it lives – it would be how everyone got notified of the content. The value is in the API, not the eyeballs.

As a global notification infrastructure, Twitter would not be able to monetize the eyeballs on the content, but they could monetize the notifications by including ads in their notification streams, and optionally requiring services to pay to not include Twitter’s ads in their streams.

Here are some steps Twitter could take to make this vision a reality:

1. Buy Gnip and Datasift — the companies they presently (and inexplicably) have handed their entire API business to. Twitter should own these companies and be the source for its own data.
2. Provide a free and premium version of their API and firehose streams. The free versions carry Twitter ads, the premium versions don’t.
3. Stop trying to own and monetize all the eyeballs on Twitter.com and official Twitter apps. Instead, do a 180 and go back to encouraging third-party developers to build Twitter client apps again. Use these apps to massively increase Twitter’s reach, traction, and monetization. Distribute Twitter into the streams of any apps that use the free API, or make money from any apps that opt out of the ads and pay for an optional premium API.
4. Sell Tweetdeck to Ubermedia (or someone else) for $50mm. That would not only be ironic and hilarious, it would be brilliant. That money could be better spent on enriching their publish-subscribe infrastructure. Twitter should be working on becoming like a TIBCO, but for the entire Internet.

Twitter has to take evasive action to increase their surface area by letting as many apps as possible integrate their API. They have to spread out, instead of fighting to be a destination. They have to stop cherry picking their ecosystem and instead enable it. Twitter’s strength is their ecosystem and their massive surface area. Without that they will be marginalized.

We’re already seeing the beginnings of Twitter marginalization happening with Google not renewing their licensing agreement to include Twitter in their real-time search results. Microsoft appears to be following suit.

Twitter’s best move to counter this is to make sure that Twitter content appears everywhere else, in every app, in every website. But they can’t do this by trying to compete with those apps and websites for the same eyeballs. Instead, turn all of them into “Twitter clients” and build a massive distributed real-time ad network.

Twitter cannot win as a destination and they are wasting their ammunition trying to do that. Facebook has them boxed in on one side and Google+ has just flanked them on the other. They have to punch through or they will be totally surrounded. But they CAN win as a notifications infrastructure. And that’s their real strength anyway.

Google+ has seen some good initial uptake from early-adopters in its first few weeks. But how will it leverage developers and partners?

In order to really build value around Google+, of course Google will integrate it with their other products, including Search, Gmail, and more. That will get it in front of a lot of people, many times a day.

But that’s not enough. Google+ has to grow as a product – it has to innovate and evolve, to stay ahead of everyone else. And it has to get hooked into other products so that it become part of everyone’s workflow even when they are not in a Google-owned site or app.

Even with all the engineers at Google, Google+ cannot innovate or integrate fast enough on its own: it has to find ways to get third-party developers to do some of this work in parallel.

The way Twitter evolved was through their API: They leveraged a huge army of free engineering talent around the Web to build them into everything, and to prototype features for them to cherry-pick into their core app.

What about Google+? Will they innovate via an API in the same way? Actually I think it may happen a bit differently.

Google+ has something unique to leverage: Chrome, a platform Google owns. While Google+ may release an API as well, I think Chrome will be more of an early focus and leverage point.

First of all, Chrome is easy to extend and it has a marketplace built in now. Secondly, by driving developers to extend Google+ on Chrome, Google kills two birds with one stone: they get innovation on Google+ and they make Chrome even more necessary and valuable – it will be the best way to use Google+.

I think Google+ is about to explode in new features, and most of them are not going to come from inside Google: They’re going to come from outside developers extending Google+ on Chrome. And this may be Google’s trump card.

Within 6 months there could easily be at least hundreds of new Chrome extensions that add functionality to Google+ and within 12 months there may be thousands of them. If this happens it could completely change the game for Google, Twitter, and their developer communities.

Most of these Google+ Chrome extensions will be features, not full applications. The Google+ product team and M&A team will be able to sit and watch to see which ones get the most usage, and the best of those will become “build vs. buy” candidates for addition to the core codebase of G+.  Think of it as a massive distributed A/B test, a decentralized genetic algorithm to evolve the best new features for Google+.

As these Chrome extensions come out from all directions, they will increase adoption and usage of Chrome, and since they won’t be available on other browsers, Chrome will gain market share. So not only will Google+ increase traction, so will Chrome, and ultimately the entire Google platform.

Many of these Chrome extensions will be free, but some will be paid apps. And Google will share in those revenues. This could be a huge driver of Chrome as a platform in its own right and could really make the Chrome Web store into a big business.

But what about a possible Google+ API; where does that fit into this equation? I think it will certainly happen, but I bet Google will focus more on Chrome extensions first because an ecosystem of Chrome extensions driving eyeballs to Chrome and Google+ simply has greater value to Google than third-party apps using an API.

A Google+ API would be great for developers and third-party businesses, and ultimately for streaming Google ads into G+ streams in third-party apps. But it will not drive adoption of Chrome. Therefore, I’m willing to bet that Google is going to delay the Google+ API for a while. That will leave developers no choice but to make Chrome extensions as a way to build on the Google+ platform. And they will. Every developer I know is drooling about this right now.

But one thing that’s important to note – Google has some work to do to really enable a massive upsurge of Chrome extensions for Google+.

Some of the developers I work with have been poring through the G+ JavaScript and data streams. And what they’ve found is that Google has not made it easy to augment their streams. The code is highly obfuscated and it’s really hard to find what you need in there.

Writing third-party Chrome extensions to augment G+ is doable, but not nearly as easy as it should be. One developer called it “a real pain to find what you need in their code.” Google should make this much easier by making their G+ code and data much more accessible, readable and extensible by third-party developers on Chrome.

If Google+ can add more transparency and developer hooks, it could be a big win for them. Google’s famous “Not invented here” syndrome may then take on a new meaning – much of Google+ may literally not be invented inside of Google, and will happen in parallel all around the Web, leading to massive parallel innovation and developer adoption.

Now what should Twitter do about this? How should Twitter respond, and what should their strategy be? I think Twitter has to change their strategy and focus on their API: Here are my specific recommendations.

I was recently interviewed by Stephen Ibaraki and Alex Lin (CEO of ChinaValue) in what turned out to be the most interesting, far-reaching, and multi-disciplinary (and long) interview I’ve ever given. I was very pleased with the depth of their questions and the topics we covered. You can listen to the MP3 version here, or read a full-text transcript here.

Topics covered:

• My work over the last few decades
• Big life lessons I’ve had
• My recent “Venture Production Studio” concept
• Stealth ventures I’m working on (realtime web, wireless power, etc.)
• Intelligent assistants
• Predictions for the future
• Augmented reality
• The Singularity
• Do we have free will? Will that change as Global Mind emerges?
• The changing nature of individuality
• The Psychological Singularity
• The Global Brain – history and implications
• The WebOS – Which cloud will win?
• The Semantic Web – what it’s really for, is it being adopted?
• What level does the brain compute at? Neural vs. quantum.
• Nature of consciousness (Buddhist view vs. Western Scientific view) – “I think, therefore I am” vs. “I am, therefore I think”
• The nature of self & possibility of artificial selves
• The Singularity
• John Searle’s Chinese Room thought experiment
• Digital physics & cellular automata; Ed Fredkin & Stephen Wolfram
• Bostrom’s Simulation Hypothesis
• Buddhist views on ultimate nature of reality
• My relationship with Peter Drucker (my grandfather) and his influence (management, knowledge workers, social sector etc.)
• The shift to a now-centric civilization
• The fragmentation of the Semantic Web
• Freeing intelligence from human brains (like we did with knowledge)
• Symbiosis; Part vs. Whole – When does the Global Brain change to a new level of order?
• Beyond Homo Sapiens – What’s next? Cyborgs, collective beings, etc.
• Technological ethics – what kind of future are we building?
• Combining the best of Asian and Western intellectual approaches
• IBM-Jeopardy Challenge

TechCrunch kindly ran my most recent article today — the full version is available here.

Here is an excerpt:

I’ve been puzzling over Twitter’s recent tactical moves around their API, Ubermedia and Tweetdeck, for a few months now, and it just doesn’t add up. In fact I think Twitter’s current strategy may take them in a direction where they end up missing out on their biggest potential win.

If Twitter continues to go down the media company path, without incorporating their API into the plan, that could not only force a large part of their ecosystem to go elsewhere, but it could deprive them of a much larger potential infrastructure revenue opportunity, and could even end up costing them the company.

After all, Silicon Valley is littered with the  burned out wreckage of once-great media companies that failed create and keep third-party app ecosystems: AOL, Friendster, MySpace, Yahoo – to name a few. It’s very hard to maintain leadership as an online media company without an ecosystem of outside apps increasing reach, innovation, and stickiness.

In light of this, I’ve been exploring an alternate path for Twitter that leverages their API in a much bigger way, and this path appears to be a better strategy. According to my own experimental revenue  projections for Twitter, this alternative path is not only a good tactical move, but it’s a good business move because it increases Twitter’s reach, number of active users, and revenues massively.

….. Read the rest here.

I’m pleased to announce that my newest venture production is beginning to unstealth. It’s called The Daily Dot and it promises to be “the hometown newspaper of the Web ” — the community newspaper for Web.

The story of The Daily Dot began several years ago when I was thinking about where the Web was headed. At the time I was thinking a lot about the future of emerging online communities such as Digg, Facebook, the early days of Twitter, and even my own Twine.com — as well as about the growth of fully immersive games and virtual worlds like World of Warcraft and Second Life. I realized that these communities really were virtual places, and some of them literally even contained the equivalent of towns, leagues, guilds, nations.

But when I looked at how the media was covering the Web at the time, I saw a huge gap. Coverage broke out into two areas: stories aimed at an industry audience (TechCrunch, ReadWriteWeb, GigaOm, Venturebeat, Techmeme) and coverage aimed at an early-adopter tech audience (Wired, Engadget, Slashdot, Boing Boing). But collectively, these audiences made up only a small slice of the overall Web audience pie.

Even more notable was that nobody was covering online communities like places — the way that newspapers cover nations, cities and towns. There were no local reporters, embedded reporters,  no stringers or correspondents in various online communities. In short, traditional media was covering the Web like a technology, not like a place.

Where was the coverage for the majority of the audience? The mainstream consumers who spent the better part of every day in this new place we call the Web? Where was the coverage of what was happening in the communities on the Web? The stories about the people on the Web? The stories for the people who used the Web?

Curiously, when I dug into this, I found that the mainstream was receiving scattered attention, in the form of only a few human interest or business articles per week in the major national media outlets (New York Times, Wall Street Journal, CBS, NBC, ABC). Mostly these articles were either curiosities or they were about big financial deals around hot companies. They too failed to address the Web as a place.

While the Internet industry audience was being deluged with thousands of geeky articles and blog posts every day, the mainstream audience was for the most part being ignored by the media. My mainstream audience member friends confirmed this – they had no clue at all about what was really going on online – even around topics they cared about like brands, celebrities, music, major privacy issues that would affect them, the birth and death of major online services, new social trends and memes, new legislation, cybercrime. The small amount of this news they were aware of, reached them weeks after it was fresh, when the major outlets finally covered it.

I realized that here was an opportunity – in fact a need – for a newspaper that covered news about the Web for the people who use the Web — mainstream people. A newspaper by and for the people of the Web (in other words, all of us). A newspaper that would cover the Web like a place and as a community. In further discussions about this concept, my wife, Kimberly Rubin, came up with the perfect name, “The Daily Dot” and I went about buying the domain name.

The idea gestated and grew. I couldn’t stop thinking about it. Finally, I decided this was a good enough idea to actively produce it in my new “venture production studio” — And so with that in mind I began looking for the right CEO and co-founders to produce the venture around.

As fate would have it, I had been introduced to Nicholas White through my longtime friend and PR guru, Josh Jones-Dilworth. Nick and I had been circling for a while. He had this “young Richard Branson” vibe — which everyone comments on after they meet him. I knew he was going to be someone important but I wasn’t sure exactly in what way.  Then it hit me.

Nick grew up in a newspaper family, working in the print newspaper biz. For over a century his family has been running community newspapers; Today they own 22 newspapers and radio stations. Like me and Josh, Nick had been thinking about the same problem — how to evolve community news reporting for the new millennium, but from the perspective of the newspaper business.

Nick was thinking about how to save the newspaper business — thoughts which he has elaborated on this week in  a new article about how he hopes to save the newspaper business by leaving it. As we spoke about the Daily Dot and his own ideas, I realized Nick had both the pedigree and the passion to build what I had envisioned. Nick was the perfect CEO for the Daily Dot. And so we invited him to co-found the venture.

With Nick’s experience at the helm, we are already making great progress. An example of this is last week’s announcement that the widely-followed editor, Owen Thomas, has left his position as executive editor at Venturebeat (a terrific publication that I read every day), to join Nick and the team as founding editor of the Daily Dot. And around Nick and Owen we are already growing a team of really awesome editors, writers, designers, coders, marketers and investors. It’s really starting to take shape, rapidly.

Owen in particular brings a strong editorial background, and is already helping to focus our strategy. We were all impressed by Owen’s incredible network of connections to the movers and shakers of the Web, as well as to the users of the Web — and also with his knowledge of all things media, pop culture, gossip, fashion, design, entertainment and more. He really understands what people use the Web for, and he’s got a great nose for news. In short he’s got exactly the right mix to head up the Daily Dot’s editorial strategy.

As Owen explains it, The Daily Dot, is going to cover the Web in a new way: It’s about people. We’re going to cover the Web not just as a technology or an industry, but as a community — actually a community of communities — spread across a virtual landscape of online places. Some of these communities, like Facebook, are even larger than physical nations, and contain communities within them that are larger than many cities. Others, like World of Warcraft, are complex parallel worlds complete with warring factions and their own economies.

And there are many other vibrant communities on the Web: Youtube, Etsy, Second Life, 4Chan, the Word Press blogging community, Tumblr, Reddit, and literally millions of micro communities around vertical interests. These communities have people in them — yes actual people, not just technologies and venture capitalists! And these people have stories, stories we want to know about. And so do the people who participate in them. But who is telling those stories?

Imagine a nation or city without its own daily newspaper — how would people know what’s going on, what would hold it together, would it even feel like one nation or city at all? A newspaper is a critical enabling catalyst that transforms a crowd into a community. It gives people news, but also a sense of place, a sense of belonging, a sense of community. It tells the story of the place, it holds the record of the place. On a deeper level, a newspaper provides a mirror of the whole back to the parts, enabling an essential feedback loop. In short, newspapers are the lifeblood of communities.

The Web today is like that nation or city without a newspaper. It’s missing something essential – the one key catalyst that will transform it from a crowd into an actual community. By providing the Web with its own newspaper, The Daily Dot will make the Web feel more human, more connected, and more cohesive. And this is really important.

The Daily Dot aims to be the community newspaper for the Web as a whole, as well as for each of the communities within it. And by doing this, we may just end up play a key role in the life of the Web. But that’s the just the beginning: We may also become the first truly global newspaper; the newspaper with the largest daily readership on the planet. After all, what newspaper today has 6 billion daily readers?

There is no geographic print newspaper audience that large. But The Daily Dot is not limited by geography; it has a real chance at achieving a truly global readership by covering the one community that everyone on this planet has in common: The Web. It’s the first newspaper that everyone may actually read every day.

The Daily Dot is still young – in fact we haven’t really even launched it yet. And as we launch it’s going to be a work in progress: We’ll be starting with a series of experiments, a newsletter, and some explorations of new approaches for involving the community in making its own news, and then we’ll be launching a major new site — currently in private beta.

Meanwhile, we’re hiring writers and editors, so if you share our passion for this mission (and you’re awesome) definitely apply. We look forward to hearing your stories.

I’m writing this post since many of my friends and colleagues have gotten wind of some news and asked me what I’m up to. This is just the first in a series of articles I’ll be writing on this topic.

In a nutshell, I’ve been working behind the scenes for the last year to co-found and angel invest in a number of exciting new ventures. Several of these ventures will be launching soon, and so it’s time to begin telling the story of what they do, and the big idea behind them: a new approach to building startups that borrows from how Hollywood produces movies.

And in keeping with this, I’ve moved with my wife Kimberly Rubin (a TV producer with 11 movies to her credit), from San Francisco to Los Angeles, where Hollywood production studios began. I believe LA is a great place to build this concept out.

I call this new model of venture incubation the “production studio model” and in this approach I work as a producer of ventures, not merely a founder or angel investor.

As well as being a better fit for the needs of early stage startups than the typical angel investor or VC approach, the production studio model has enabled me to start a number of really excellent ventures, for less cost, in less time, than I ever thought possible.

Production Portfolio

But before I go into more detail about the model, here is the current portfolio of companies that I am actively producing. With the exception of Live Matrix and Klout (which were started earlier), all of these companies were started in the last year and will be launching soon:

• Live Matrix — The schedule of the Web. Live Matrix is the only guide to what’s happening, when, online – across all media types (video, audio, chat, gaming, shopping, and more). I co-founded and seed-funded this venture with CEO, Sanjay Reddy, and I continue to incubate it and actively participate, in the same way as I have since we began, by serving on the board and helping on product strategy, marketing and technology. Live Matrix is launched and busy making deals and launching new features. More news coming soon.
• Klout — Klout is the standard for measuring influence. I discovered the company when I was a judge at the SXSW Accelerator in 2009. I was really impressed with the founders and soon became the company’s first outside investor. Since then I have served actively as an advisor to the company. They recently raised a terrific venture round with Kleiner Perkins and are off to the races.
• Bottlenose — You’ll be hearing a lot about this venture soon. It’s a new personalization technology designed for the realtime Web and social media. I co-founded this company with Dominiek ter Heide. As well as seed-funding the company, I’m taking an increasingly active role in helping to build this company.
• The Daily Dot — The Daily Dot is a new online newspaper about the Web, for consumers. Most of the coverage of the Web today is targeted at the tech industry, a tiny fraction of the audience, but the Daily Dot will cover the Web for the majority of the audience: consumers who spend much of their day, every day, online. The Daily Dot hasn’t launched yet but it’s going to be an exciting company. I co-founded it with newspaper-industry CEO, Nicholas White and co-founder Josh Jones-Dilworth. More news will be coming out soon!
• StreamGlider — StreamGlider is a new visual real-time dashboard for tracking interests across various types of devices, starting with the iPad. It’s got a gorgeous user-interface and some novel features that are especially suited to keeping up with streams of rich media. I co-founded this venture with two leading technologists in the information filtering and semantics space, Bill McDaniel and John Breslin. This will be launching soon as well. More news will be available at launch.
• “Project Nikola” This new venture has a breakthrough new energy technology and is totally in stealth. This isn’t even its real name; even that is a secret, for now. What I can say currently is that it really works, it’s mind-blowingly cool and just may disrupt the entire power grid someday. But there’s still a lot of R&D to do before we release it.
• The Earth Dashboard. This is a not-for-profit initiative (yes, I sometimes help produce game-changing nonprofits too) that is working to create an interactive live dashboard about the state of the planet that brings together, and visualizes, all the key global indicators, in one place for the first time. This project is led by Medard Gabel, who worked with Buckminster Fuller, and the creative director is Mia Hanak and her accomplished museum exhibit design team.The Dashboard will be available online as well as in major physical public locations around the globe.

Several of these companies have a common thread – and a common passion for me — they are focused on helping people filter the Web in potentially disruptive ways. Some are using “Big Data” analytics, data mining and extraction, natural language processing, machine learning and semantics, to understand the Web. These are areas that I am deeply familiar with from my many years working around information filtering, AI and search. The Nikola project is an exception — it is outside of the Internet space but springs from a multi-decade interest I’ve had in radical alternative energy technologies.

A History of Incubation

Since 1994, I’ve been involved in starting companies as an entrepreneur, and since 2000 I’ve also been an angel investor. Through incubating numerous ventures (my own and those I’ve angel invested in), I’ve gained some experience into the art of incubating startups.

But one of the best experiences I had was starting one of the more successful incubators, nVention, at SRI, which I conceived of and co-founded with Norman Winarsky (now head of ventures at SRI) in 1999. nVention is now global and has launched more than 40 ventures.

Unlike many incubators, nVention acts in a very hands-on way. First of all, most of nVention’s work focuses around creating ventures to commercialize intellectual property that was originated at SRI. Secondly, nVention bring teams of internal and external experts together to help incubate its companies from concept stage through commercialization. In effect, nVention acts like a production studio, and the people who work there function like producers. It’s a model I’m emulating, albeit in a more grassroots and distributed way.

The Venture Production Model

As a producer, I work actively to develop new original intellectual property, or to source it from great innovators, and then I angel invest and/or bring funding to the deal, shape products and strategies, build teams, invent and develop products, and actively grow companies and take them to market. In many cases, the ventures I’m producing are originated by me, but I also have several ventures in my portfolio that were originated by others, or in partnership with others.

The key to this approach is that I usually get involved at or even before concept stage — even before there is a real team — and I actively work to shape it into a venture, from concept through commercialization.

To accomplish this, across more than one venture at a time, I partner with excellent people to co-produce these companies. In some cases I act as the startup CEO, in other cases my partners do, or we find and partner with the right person to be CEO. Often I find myself partnering with entrepreneurs and helping to coach them to be CEO’s. But in all of these cases, we focus on producing ventures together, as a team. And we’re all in it for the long-term. Because I’ve found excellent partners to work with, including excellent CEO’s where needed, I’m personally able to focus more intensively on helping each venture. This has worked very well so far.

The production approach to venture creation is quite different from the “fire and forget” or “spray and pay” (or “pay and pray”) model that many VC’s and angel investors are engaging in. Instead of spreading lots of fairly hands-off bets across dozens of companies, in the production model I really focus and get deeply hands-on with a pipeline of projects of various stages.

This the opposite of the index fund or hedge fund approach that some funds are taking in the Valley. And I think it is a much better fit for the needs of early stage companies.

The production approach is also different from what many incubators and start-up accelerators are doing these days. Incubators and accelerators play an important role in the startup ecosystem, but the key difference is that in the Hollywood-inspired production model I’m testing, I often start earlier in the process – before there is a concept, company, product, CEO or even a team. Many incubators and accelerators start later in the process.

Here’s how it works. My associates and I source candidate ideas both from our own stream of inventions, as well as from people in our network, and from other innovators we find or who find us. Next we “option” the best ideas with joint R&D agreements and/or with initial prototype funding to test them out. We then filter these prototypes and choose the best ones to produce.

We then form companies, build teams, develop business plans, branding and strategy, and bring additional funding together to develop the commercial offerings, launch, market, and grow them into full ventures. It’s a very hands-on process – and just like movie production, and we usually have a number of projects going on at each stage in our pipeline at once.

The process is similar to producing a film. With a film, first you have to create or find the story, then hire writers and a director, recruit talent, and build a production team, get the financing and early distribution deals in place, shoot the film, do post-production, get broader distribution, market the film and release it.  In the early stages of companies, we all wear many hats, and as they grow, we specialize. In my own case, I assume different roles and levels of involvement according to the needs of the ventures as they grow.

The New Role of the Producer in Tech

A key to this process is really in understanding the new role of a producer in the venture world. It’s not exactly the same as the role of an angel or VC, or an EIR, or even a typical “superhero CEO.” It’s a new role that connects them all together.

Another key is having a model that is designed to attract excellent producers and talent to team up with. To do that, I’m working with a structure that gives my partners a better opportunity than they can find anywhere else. Whether it is an externally or internally originated venture, the model I’m working with is, by far, the most entrepreneur-friendly deal in the entire industry.

In most of my ventures today I take a minority, or at most, an equal partner position, with my cofounders – even in the ventures I originated and funded. In some of the ventures I have angel funded, I have continued to maintain the original equity split with my co-founders, even as the amount of funding I have contributed has increased over time. Where other angels and VC’s take the approach that money is everything, I take the exact opposite view. Talent is everything. Talent is rare, and it’s the lifeblood of ventures.  I don’t believe it is healthy for any company to have the investors take control away from founders — too often that results in disaster. My model is all about cultivating and facilitating the founders.

Why do I do this? Because I believe that you get the best out of people when they really feel they own their venture, and when they feel respected and valued for their contributions. Part of this is because as an entrepreneur myself I have experienced life on the other side. I know intimately what it is to be an entrepreneur, and I know how some VC’s, and even some angels, take advantage of entrepreneurs and founders, suck the life out of companies, and destroy businesses by over-controlling them, and I vowed NOT to be like that.

The terms I offer are the same terms I always wanted for myself. No bullshit and no games. We all succeed or fail together. It’s a true partnership. I’m not betting the odds across dozens of companies and expecting only 1 or 2 to survive; I’m NOT doing shotgun investing like most angels out there, I’m making extremely careful, extremely deeply involved long-term commitments to build companies side-by-side with my partners. And I don’t just talk about this, I do it – my model reflects this.

At the same time, I also spend money in a different way than angels or VC’s. For example, in many of my projects I’ll start on spec with a developer and an idea. No money is initially invested. They work on the idea and prove they have the goods. Then if that works, there is a small grant to “option” it and we develop it further, much like a production studio options a story. I then work with the technical or product teams to see what they can deliver with this initial grant – usually a prototype. This is a test.

If the test goes well and progress is good, we make a production deal in which money, and time, are invested in stages, by myself and others in my network, as needed, rather than all at once. We’re not talking about huge dollar investments early-on, it’s frugal and careful, but it’s enough, and it’s extremely value-added. Later in the process, when the time is right, more money can be brought to work.

Building out the Production Studio

Another way I help my portfolio companies is by bringing pre-negotiated deals with handpicked best-of-breed vendors for many of the services they need. I bring the top law firms and patent teams, the best PR and marketing teams in the business, and accounting and HR services, from partners I know and trust. This saves ventures valuable time and money, and protects them from making early mistakes.

Once the ventures are aimed at a clear target and have something to show, we go together to other angels and venture funds in my network to raise the roll-out money. There’s no reason to give up equity until we need to. The result is that entrepreneurs who work with me end up owning bigger stakes of their ventures than they would if they worked with traditional angels and VC’s.

On the investment side, I’ve been meeting with interesting angel investors – some are pros and some are new to this – and we’re teaming up to jointly fund these ventures together. By working with a production team such as mine, angel investors can put their money to work with less hands-on effort on their parts – because we’re doing the production work for them. That doesn’t mean they aren’t involved – we consult with them as much as they want and we actively solicit their feedback and ideas at every step — but it means they can have confidence that an experienced team is doing the groundwork day-to-day. By working with producers like myself and my team, angels and funds can spread their bets without being spread too thin. If you’re an angel investor, or even a VC, and you’re curious about working with our network, drop me a line.

On my own side, of course, I have to be very picky about what ventures I get involved with, so that I and my team are not spread too thin as well. For that reason, we only allow for about 4 ventures in any stage (from concept stage, through R&D, to beta, and commercialization) of our pipeline at once. But this is a model that I think can scale as we add more producers and team members to the mix. Scaling this is an area that I am actively thinking about right now. If you’re an exceptional venture producer and you would like to be involved, get in touch. This is not about anyone being a superhero, it’s about creating an awesome and highly collaborative team, supported by an incredible network.

There’s so much more I could say about all this, and in time, I’ll write more about it.You will be hearing a lot more about this model, as well as all of the ventures we’re producing in coming months, as well as some new ventures not listed above that are in the pipeline and will become visible in the future.

In the meantime, I’ll be posting here and on my Twitter feed, about my thoughts and what we’re learning as this goes forward. I welcome your thoughts too — this is one of the reasons I’ve written this. So please don’t hesitate to ask questions, offer advice, or observations. Stay tuned, it’s going to be an adventure!

– Nova

My father, Mayer Spivack, passed away on February 12, 2011, in the Kaplan Family House, a beautiful hospice outside of Boston. He passed away, at the young age of 74, after a difficult year and a half battle with colon cancer. During his illness he never lost his spirit of childlike curiosity, enormous compassion, and his dedication to innovation.

His passing was at times difficult, but ultimately peaceful, and took place over five days, during which he was surrounded by love from close family and friends. His presence and spirit, and the intense experiences we all shared over those last days with him are unforgettable: the most incredible experience of love and spiritual connection I have ever had. He was as great in death as he was in life.

This is the story of my relationship with my father: the things I appreciated most about him, what I learned from him, and what he gave to me at the end of his life. By sharing this, I hope to amplify and share his gifts with others.

My father was a truly unique person, and a Boston legend. He was multi-talented and worked in many fields at once, mastering them all (you can read more about his actual work here). He had a vast intelligence, a palpably original approach, and an even greater heart. He was a true Renaissance Man, a great intellectual and artist, and often an unintentionally entertaining and eccentric genius. He had a profound influence on all who knew him well.

As a father, he was a large, warm, loving, fuzzy bear of a man who never really lost his childlike innocence. He was the kind of father everyone wanted to have and when they met him they instantly wanted to hug him. His greatest accomplishment was his compassionate heart: Everyone could feel it.

But despite his brilliance, or perhaps because of it, my father never really fit in. There was no box that could contain him. He was an only child, a loner, and an outsider with little interest in conformity. He had a disdain for formality and social conventions, which always manifested, much to our embarrassment, in the most formal and conventional of settings. He described himself as an iconoclast. Despite his unconventional ways, he was loved and appreciated for his humor, his quirkiness, his unselfconscious originality, and his always out-of-the-box thinking, even (and sometimes especially) by those in the mainstream.

One funny story we recently remembered illustrates his irrepressible spirit: He was invited with his wife to a major European conference of art restorers in Italy. There was a formal reception at the home an Italian Duke. My father, never comfortable with any kind of formality, playfully took one of the candles from the reception, and wore it on his head for the entire night. During the 5 course formal dinner and the reception, he was introduced to various members of the Venetian nobility and the European art world, all the time, balancing this burning little candle on his head, yet also acting completely as if it wasn’t there and not acknowledging it at all. Everyone thought that, because of his first name, “Mayer,” he was actually the eccentric “mayor” of some city in the USA and so despite their horror they were too afraid to point out that there was a candle on his head.

In another infamous incident, my father sat on the Arts Council for the city of Newton, Massachusetts. One day a photo was taken of the Council members, none of whom were actual artists, aside from my father — they were prominent upstanding Newton business leaders and socialites. In the photo they are all wearing three piece suits and looking very formal and proud. My father is also wearing a three piece suit, except that, much to the dismay of the other Council members, his suit pants are tucked into gigantic calf-height silver moon boots (to him it was winter and it was perfectly logical to wear snow boots).

In a similar vein, whenever my father was invited to a black tie event, he would reluctantly attend, dressed appropriately, except with a black dress sock tied around his neck instead of a bow tie. Of course he would never acknowledge this to anyone, and they were all too shocked to point it out to him.

One more example of my father’s individuality: when we were children in the 1970′s in Boston, my father got a great deal on a World War One field ambulance. That was our family “car.” He also had a longstanding love affair with army surplus, to which he had special access through his position on the faculty of Harvard Medical School. From some special warehouse, he acquired a full Coast Guard extreme-weather helicopter rescue snowsuit — a bright orange practically bulletproof insulated monstrosity. To him it was extremely practical – warm, waterproof, and visible even in the worst white-out snowstorm conditions.  He was entirely unselfconscious of the fact that he looked like he had just descended from a rescue helicopter when he wore it. And so this was what he wore, along with his usual silver moon boots, all winter, every winter, through my early childhood.

My poor brother and I would have to be dropped off every morning at elementary school this way: We would pull up in an an antique white ambulance — a big man in an orange emergency jumpsuit, sunglasses, and silver moon boots would get out, tromp through the snow, and open the rear doors (where the stretcher would normally be) and then my younger brother and I would pop out, much to the shock and awe of our fellow schoolmates. Thus were the origins of my own life as an alien and outsider. While these experiences were a source of horror and embarrassment for us growing up, today we laugh hysterically when we remember them — they are what we are made of and I wouldn’t trade them back for anything.

My father was a huge influence for me as an innovator. He was a prolific, constant professional inventor and my childhood was filled with his inventions, in various stages of development. He was such a good inventor that corporations like Polaroid, Otis Elevator and others, would hire him to come up with inventions. I remember him once telling me that he made 100 inventions for Polaroid in 100 days. There was another time when my father was hired to invent new uses for Silly Putty — he received a giant vat of the stuff from the Silly Putty people. With the attention of my father, two kids, and all our friends, the Silly Putty gradually dispersed throughout our house, until little blobs of Silly Putty could be found in every corner, crevice, crack, cranny and nook.

My brother and I grew up inventing things with our father. In fact, we were not allowed to have or watch a TV as children – instead we had three rooms dedicated to making things, in which we spent most of our time: one for building things with wood, one for drawing and painting, and another was my father’s studio. These rooms were stocked with all kinds of tools and art supplies.

As an inventor, my father always had tools and various devices hanging off of him, clipped onto his belt, in fanny packs, in holsters, backpacks, special cases, and in holders of his own making. Our nickname for him at times was “Inspector Gadget.”  He was always infatuated with some new tool or device.

I remember, for example, what we refer to as his “Hot Glue Phase,” when I was in junior high school. Hot glue is a plastic that you melt through a device called a hot glue gun. It creates a white plastic goo that hardens as it cools and is unfortunately able to fasten just about anything together, much to my father’s delight, and our misfortune. I remember going to junior high school with a rip in my pants repaired visibly with hot glue, my sneakers repaired with hot glue, my book bag repaired with hot glue. There was nothing that hot glue couldn’t be used on, we discovered. Clothes. Plates. Furniture. Our house was at one time filled with little spider web strands of hot glue residue, stringing together our possessions, our home, our clothes, us.

One of my father’s most memorable inventions was “The Body Sail” – a precursor to the Windsurfer, on which the sail was not attached to the board  but rather was held by hand using a special boom. He once won the Charles River Boat Festival sailing that contraption – of course, wearing a full body scuba suit. My brother and I used to use his Body Sail on ice skates in the winter, on frozen ponds. My father, of course, preferred to sail it on roller skates, in full bodysuit, helmet and gloves, right through parting waves of startled lunchtime crowds in Harvard Square.

No story about my father would be complete without mentioning his love of sailing. It encompassed not only his Body Sail invention, but a series of boats, particularly multi-hulled boats such as catamarans and eventually trimarans. In his later years he moved to Marblehead outside of Boston, a worldwide center of sailing, where he became an avid fan of high-speed sailing, eventually designing and starting to build his own trimaran out of aerospace composite materials, which, had it ever been finished, would have been among the fastest, and certainly the most computerized and advanced, trimarans on Earth.

My father was also a classically trained artist and particularly a widely shown sculptor — I grew up surrounded by his artworks — photos, drawings, and sculptures made from found objects, industrial artifacts, natural materials. I played in his studios – surrounded by tools for making things, prototyping, and inventing. As an artist, my father was also truly unique. An early pioneer of the use of “found objects,” his artworks were made from rusty pieces of industrial machinery, wooden molds for casting pieces of ships, old rusty farm tools, pieces of found wood and materials from nature. I grew up surrounded by these artworks. There were hundreds of them and he had numerous exhibitions.

One series of works he called “Foundiron” consisted of pieces taken from the intestines of large industrial boilers and furnaces. Another series used wooden molds for casting brass for ships, appeared like a set of primitive human figures – perhaps from Easter Island. Later works included a two ton angelic shape made from the massive steel blades of a snowplow for train tracks, and gossamer drawings in air made from the unwound springs of massive clocks that reminded one of Picasso’s drawings. His Shrine Series included animal bones, bird wings, industrial spindles, parts from clocks, early computers, and metronomes, and melted industrial alloys. One of his larger installations is made from three giant steel train car hitches that he cut apart and welded back together like hands grasping each other, and now stands permanently in Boston’s new South Station.

He was also a photographer and some of his images — for example macro images of honeycombs and turtles, still remain in my mind as if I saw them yesterday. At one point his entire office was rigged up with a complicated system of prisms, blackout shades, lenses, reflective materials, and rear projection screens so that he could take photos of shapes made of pure light that he called Lumia – which he then blew up to massive size and animated with a bank of slide projectors — some of these images can be seen on his weblog.

Another area of life that my father dove into deeply was music. He had a profound connection with music. His music collection included many of the greatest works of classical music, but also Jazz and folk music, and even Indian classical music. Our childhood was filled with music, and also with musical instruments of all kinds – particularly unusual instruments: aboriginal instruments, vibraphones, banjos, harpsichords, flutes, guitars, percussion instruments. My own broad taste in music came from this. My brother, Marin Spivack, took it even further, becoming a masterful Jazz saxophone player, as well as learning to compose for and play guitar, drums, piano, bass.

My father’s fascination with science and his massive appetite for knowledge translated into a home filled with books about science, scientific journals, and discussions about physics, biology, chemistry, brain science, psychology, architecture, engineering, and anthropology. We spent countless hours discussing science, the future, the brain, and technology, and coming up with new theories and inventions.

In my own life as an innovator, my father was my biggest fan and supporter. He taught me to invent – it was his passion. He wrote about it, and refined his theories and methods for innovating and enhancing creativity over the course of his life, and as children my brother and I were his very fortunate experimental guinea pigs.

I can remember being brought by him as a child to MIT, the Massachusetts Institute of Technology, where my father had done his graduate studies — there my brother and I were subjects in early experiments on children and computers: we were observed as we played the early computer game, “Wumpus,” and learned how to use computers, by his colleagues. I still remember my father’s love for MIT — how he took my little brother and I on nighttime expeditions into the hidden catacombs under the campus, and the many times we met with his friends, colleagues and relatives from various MIT departments. My father wore his MIT ring proudly right until his last breath: It was the only club he ever wanted to belong to.

As I got older my father shared with me his work with architects and designers, and his “Design Log” methodology for documenting and improving any kind of design process. Later, as an adult he shared his new theories about human intelligence, learning disabilities, dyslexia, and what he called “syncretic associative thinking.” His theory of syncretic cognition proposes that there are two fundamentally different, yet complementary, forms of human intelligence — linear and syncretic. According to my father’s thinking, syncretic thought is associative and seemingly chaotic, yet out of it great creative leaps and innovations are born.

Dyslexics, of which my father was one, were examples of the extreme case of syncretic thinking: despite difficulties with linear logic, dyslexics are often brilliantly creative; in fact many great geniuses – especially artists, but also scientists — have been dyslexic. My father believed that instead of viewing dyslexics as “learning disabled” they should be viewed as “creativity enabled” and trained and taught differently, to leverage their unique cognitive abilities.

Instead of being viewed as bad at math or slow at reading, dyslexics might instead be viewed as unusually talented at associative thinking, brilliant in the arts and inventing. It was all a matter of perspective. My father advocated passionately for the often-overlooked talents hidden within dyslexia in his own writing, and also in his parallel career as a trained psychotherapist working with hundreds of people, especially learning disabled people, engineers and artists.

My father’s interest in the many flavors of intelligence extended not just to humans but also to animals: He had a long fascination with animal intelligence. His homes were always filled with animals – particularly highly intelligent parrots of various breeds, with whom he would speak, whistle, sing, and explore his theories about learning and cognition. When I was just a newborn, he had a pet crow — which he said was one of the most intelligent of birds.

My father painstakingly studied crows and eventually learned how to mimic their various kinds of calls. I can distinctly remember how, throughout our entire life together, he would suddenly start embarrassingly screeching, “Caaah  caaahhh Caaaaaaahhhh,” whenever he encountered a crow in some random tree.

In another famous story from my father’s MIT days, he became fascinated with echolocation — the form of navigation through sound used by animals bats and dolphins. Bats in particular became a bit of an obsession for my father. Bats navigate with high frequency clicks. These clicks bounce off of surfaces like walls, buildings, plants, insects, other bats and the reflections are turned into images in the bat brain.

My father decided that bat echolocation would be a great way to help the blind navigate through cities. So he invented a bat clicker device you could wear on your head. It would emit rapid loud clicks that were within the range of human hearing. He spent a week blindfolded, wearing this device, walking around the MIT and Harvard campuses, and apparently he was able to navigate successfully with it.

He recounted that after many days of using this contraption, blindfolded the whole time, his brain adapted and he was able to discern the different types of materials, objects and surfaces from the subtle differences in sound reflections. He was able to cross streets, navigate around buildings and obstacles, and could even find his way through crowds (although we all suspected the crowds were probably parting of their own volition around this strange blindfolded man with the clicking machine on his head). The astonished people of Cambridge who encountered him must have thought he was some kind of alien exploring a strange new world. And one can only wonder what the bats themselves must have thought.

At various times in my childhood my father also had pet frogs, lizards, turtles, fish, snakes, squirrels, cats, and later, his beloved pug. We grew up with enormous aquariums, terrariums, and aviaries — as kids these were wonderlands. This love of all kinds of living things would eventually guide him to his second wife: Boston artist, Louise Freedman. We knew they were made for each other when, for their first date, they chose to go to a local cemetery pond to collect pond water and frogs together.

As their lives merged, so did their always increasing menagerie of animals. And gradually there was less and less room, or time, for humans in their house. During my college years, my father and his wife had started raising African Grey parrots, and had also become close friends with Harvard/MIT animal cognition researcher, Irene Pepperberg, and her famous parrot, Alex.

When I would visit their home on school breaks, the parrots were as much a part of the family as my brother and I, and occupied a central location in the family room. A typical mealtime conversation in our family was a combination of English words, chirps, clicks and whistles, spoken by humans and parrots alike. My father and Louise eventually moved into a home that literally was like a tree — surrounded by trees on many levels, on the edge of a huge nature sanctuary on Marblehead Neck. There amongst the branches, they could almost live as birds. My brother I joked — half-seriously — that for an upcoming wedding anniversary, we would throw out their couch and instead replace it with matching human-sized perches for them.

But my father’s fascination with animals wasn’t just about intelligence, it was also about love. I remember one day as a child, while frantically evacuating from Cape Cod ahead of a fast oncoming hurricane, my father suddenly backed up miles of panicked traffic when he stopped the car in the pouring rain and lightning to scramble around on his hands and knees, risking his own life, to rescue a turtle that had strayed onto the freeway. This deep love of animals, and people, that he manifested throughout his life, was at times a source of embarrassment for me, but later became what I admired most about him. For my father, this simple love of all living things was his religion. But for most of my life, I didn’t realize what an accomplishment that was.

Although my father influenced me in so many ways, the most important facet of life that we shared — and struggled over — was spirituality.

He was a dedicated scientific materialist and rejected superstition, which to him included all institutionalized forms of religion. He even sometimes referred to himself as an atheist, although I think more accurately, he was an agnostic. I on the other hand, while also deeply interested in the sciences, had come to the conclusion that science alone could never fully explain reality or consciousness — I felt that there was a common underlying truth in all the great religions which science had so far completely missed, a truth that was essential for a complete and accurate understanding of reality. This debate between science and religion became the fulcrum on which we wrestled endlessly and in many different ways.

I had always known, even as a child, that there is something more than meets the eye about reality that is extremely subtle, yet at once vividly evident. Growing up, I had a number of spontaneous mystical experiences that I could not explain, and later I witnessed highly unusual phenomena taking place in monasteries in Nepal and India that convinced me that there must be more to the mind, and to reality, than our western scientific worldview could presently measure or explain. I was perplexed by the apparent incompatibility of these experiences, and the Western scientific framework that my father and I both lived and worked in.

In my attempts to reconcile these two worlds, I became obsessed with physics, computer science and artificial intelligence. I began searching for a grand unified theory. I sought to create software that could simulate physics, the brain, and the mind.  With some of the world’s most cutting-edge physicists and computer scientists, as well as at some of the top artificial intelligence companies, I worked on on several major initiatives in computational physics, parallel supercomputing, and artificial intelligence, as well as my own software projects and theories.

All of these attempts failed to achieve their goals so thoroughly and so repeatedly that eventually I began to question if it was even possible to do. I reached a point where I began to doubt the assumptions behind these projects — I began to question my own questions. This led me to a deeper exploration of the mind and the foundations of reality – a journey from cognitive science and physics to philosophy, and finally to spirituality. Paradoxically, I ended up back where I began, looking inwards rather than outwards, for the answers.

My quest for spiritual meaning took me through a survey of all the major Western and Eastern religions, and while traveling in Asia for a year after college, I landed in Tibetan Buddhism, with its intense focus on the nature of mind and consciousness. I was home. For me, Tibetan Buddhism had the perfect combination of rational and objective logical analysis (my father’s influence), and the mystical direct experience of the union of consciousness with divinity that I had tasted in my own experience.

In Tibetan Buddhism I finally found a rational yet holistic framework that could account for all the dimensions of observed experience: both the outer physical world and the inner dimensions of consciousness. From the Buddhist perspective, we humans are manifestations or projections of a deeper ultimate nature of reality, as are all sentient beings, and in fact all animate and inanimate things. This deeper level of reality is the origin of both the subjective and objective poles of experience, and it’s nature is transcendental, empty, yet aware.

The direct proof and experience of this can be found many ways: through logical reasoning, through prayer, through love, through nature, through art, through meditation, and perhaps most easily, by searching for the source of one’s own consciousness. Consciousness is a unique phenomena that we all have direct, equal, and immediate access to, yet which science cannot measure let alone explain. By persistently searching for the source of our own consciousness, and discovering that we can’t find it yet it is not non-existent, we are inevitably brought to a direct realization of the ultimate nature of reality.

Over decades of searching for consciousness, first through science, then through Buddhism, I had come to the conclusion that rather than consciousness emerging from the brain, it had to be the other way around: All experience, and indeed the body, brain and even the physical universe, emerge from consciousness. I had discovered that consciousness is a gateway to a sourceless, deep and endless wellspring of mysteries. And more importantly, I had found what I thought would be conclusive evidence that would finally convince my father that I was right.

But when I tried to relate these realizations to my father, he was entirely unconvinced. He argued that my experiences were not really objective, and that consciousness is an epiphenomenon of the brain; a wonderful side-effect, a remarkable illusion that nonetheless could be reduced to neurochemistry and atoms. I countered that in the special case of consciousness, subjective observations could in fact be objective, under the right circumstances. I claimed that it was possible to scientifically and objectively observe consciousness by looking at it under the microscope of carefully trained meditation. But he cast doubts on these claims, citing numerous examples from psychology and neuroscience.

So I tried many other arguments. I cited the work of philosophers like John Searle who provided many illustrations of how conscious experiences could not be reduced to the brain or any kind of machine. I used lines of reasoning from Buddhist logic. I even cited recent findings in quantum theory that seem to imply that the act of conscious observation interacts with experimental results. But all of these arguments failed to convince my father that consciousness was fundamental or irreducible. He remained a skeptic and I felt invalidated. And so I strived even harder to find a way to map my experiences to his worldview, so I could finally prove the scientific foundations for my experience and belief in divinity to him.

This ongoing debate between my father and I — between science and religion — was not unique to us; it had been going on for millennia, and yielded many great works of both science and art. Our conversations were often frustrating and ended in exhaustion and exasperation, but we also sensed that somehow we were getting somewhere, if not mutually, then at least as individuals. We were foils to one another, worthy opponents. Like many who had come before us, the dialectical process of trying to convince one another of our conflicting views of reality, caused us to generated volumes of new writing, theories, inventions, and ideas we could not have arrived at on our own.

Nevertheless, despite my father’s strong rebukes of superstitious belief systems, and his skepticism towards my Buddhist beliefs, he was in fact a deeply spiritual man, in a very human, unembellished way. His spirituality was not tied to any system or institution — it was natural and basic: it was how he lived and the ideals he lived by: Love, Science, and Art. His spirituality was not about words, it was about actions. He expressed it in his art, his good deeds, his compassion, his joyful creativity, and his ability to love and be loved.

What I failed to see was that my father’s spirituality was immensely humble. So humble that he would not even claim to be spiritual, and certainly wouldn’t go so far as to conceptualize it. Instead, he was simply a truly good man, a mensch. While I continued to try new tactics in my campaign to convince him, and as I judged him as closed-minded and non-spiritual, he was in fact actually living my spiritual ideals better than I could understand at the time. But, not realizing this, I was certain he was missing out on something of vital importance, something that I had to convince him of before he died. And so our debate continued.

Then, in the last few months of my father’s life, we were finally able to bridge this divide. As his illness progressed, his wife called me and urged me to visit before it was too late. “He’s really getting worse, and I want you to have a chance to be together while he’s still strong enough,” she said. And so I flew to Boston and we resumed the debate.

Perhaps it was our mutual sense that time was running out, or perhaps it was that we had both exhausted all our prior arguments, but this time we reached a level of discourse that was essentially mathematical in nature; pure logic, pure set theory. Without imposing the assumptions of either science or religion, we started anew from first principles and through pure reason and observation, we derived a new common language, on neutral ground. And with this in hand, we arrived at a single nondual phenomenology — At last we had arrived at the basic nature of reality.

When we finally reached the point of agreement and mutual understanding, after decades of debate, and we both witnessed the simultaneous unification and transcendence of our prior belief systems — we saw that we had always actually agreed on a deeper level. And on that December afternoon, as we sketched out the full picture together, in a way that neither of us had done before on our own, we both breathed a sigh of relief. It was an incredibly cathartic moment for both of us.

At the conclusion of our decades long debate, we sat quietly together, just being in that understanding — a meditation on awareness and knowledge, on physics, time and space — on our mutual respect for the immensity and majesty of the universe. I will always treasure that time.

The day after that experience, before I left to return to California, I sat by my father’s bed. He was almost unable to walk at this point. As I said goodbye, thinking I might never see him again, I said, “Don’t forget what we discovered together, it is the highest realization.” He replied, “There is still one more realization that is higher.” Surprised, I asked him, “What?” He answered, “To live it!”

About a month later my wife called again. “He’s dying,” she said, “come back as soon as you can.” The cancer had advanced unexpectedly fast and so I flew back to be with him one last time.

I stayed by his side, looking into his eyes, talking to him, even though he had lost the ability to move or speak. His eyes smiled back. My brother and I kept telling him, as he labored to breathe for the final two days, “It’s ok to go now, you can let go, we love you, we’ll be ok, we’ll take care of each other.” But his drive to love and protect us all was so strong. He wasn’t ready to go. Even while in the depths of his own suffering, he was still filled with compassion, he was worried about what would happen to all of us. It was noble and beautiful to witness.

We played him the music he loved, the music he played for us as we grew up. We laughed and told him our memories and stories of him. We stroked his hair and his beard and tried to make him as comfortable as possible as he lay there, struggling, and probably frustrated that he couldn’t communicate, and at times in terrible pain. Yet through great effort he still found ways to let us know he heard us, loved us, and was still conscious.

As his breathing changed and we saw the signs of death advancing further through his body, he maintained his clarity and brilliance and even got brighter — we could feel his heart, and see his kind and intelligent spirit in his eyes. He tried to speak to us by making what little sound he could and moving his eyebrows in response to us. “Remember what we talked about, what we realized,” I said to him over and over, and I could see he was living it.

Finally, on the evening of February 12, 2011, he let go and died peacefully in his wife’s arms as she sang to him gently. All of us felt at that moment an incredible, all-embracing, boundless love and bliss, even as we grieved. It was him. My father, Mayer Spivack. Our Buddha. He went into Love.

Tonight I am pleased to announce that my next Big Idea has launched. It’s called Live Matrix and I invite you to come check it out.

Live Matrix is the schedule of the Web — We help you to find out “What’s When on the Web” — the hottest live online events happening on the Web: concerts, interviews, live chat sessions, game tournaments, sales, popular Webshows, tech conferences, live streaming sports coverage, and much more.

It’s like TV Guide was for TV, but it’s not for TV, it’s for the Web. There are all kinds of things happening online — and while Live Matrix includes a lot of live streaming video events, there is much more than just video in our guide. Live Matrix includes any types of scheduled online events — but we don’t include offline events — to be in Live Matrix an event must enable people to participate online.

The site combines elements of a guide, a search engine, and a DVR, to help you discover events and then get reminded to attend them, or catch them later if you missed them.

The insight that led to Live Matrix was that the time-dimension of the Web is perhaps the last big greenfield opportunity on the Web. It’s an entire dimension of the Web that nobody has made a search engine for, and nobody is providing any guidance for. Nobody owns it yet — it’s a whole new frontier of the Web.

There are millions of scheduled events taking place online every day. Some of these events are very cool, some are very relevant — but there is no easy way to find out about them. To find out what’s happening when on TV for example, we have TV Guide, but there is no equivalent for finding out what’s happening when on the Web.

In my own case I kept finding out about cool online events that I would have participated in — concerts, conference streams, webinars, online debates and interviews, and sales –  if only I had known they were happening. I think many Internet users have experienced this.

Google, Yahoo and Bing all focus on what I call the “space dimension” of the Web — they help you find what’s where — where is the best page about topic x? — But they don’t help you find out what’s when — what’s happening now, what’s coming next. They only help you find out what’s already finished and done with. How do you find out what’s happening now? How do you know what’s upcoming?

It was an “aha moment” when this all became clear — there is a new opportunity to be the Google or Yahoo for the time dimension of the Web. Or at least to be the equivalent of a TV Guide for the Web.

Furthermore, All trends point to this being a big opportunity. The continued growth of the realtime Web (Twitter, etc.) and the emerging Live Web (video and audio streaming) has been discussed extensively in the media; most recently comScore reported nearly a 650% increase in time spent viewing live video online.

So with this opportunity clearly in mind I set about looking for a co-founder who would be the right person to team up with, someone who would be the CEO.

That person was Sanjay Reddy. Soon after I met Sanjay it was clear to me that he was the exact right guy to partner with: his background in media and technology were what impressed me (for example, he was head of corp dev, strategy and M&A at Gemstar-TV Guide, where he led the $2.3 billion dollar sale of the company to Macrovision, and he had also worked at other Silicon Valley startups and investment banks as well).

Sanjay and I spent quite a bit of time just talking about ideas and eventually decided to join forces. My Lucid Ventures incubator, along with Sanjay, seed-funded the new venture and named it Live Matrix, to go after our mutual vision.

Soon after Sanjay joined we were fortunate to be joined by our two highly experienced colleagues, Edgar Fereira (formerly VP of data for TV Guide Data and TV Guide Online) and Tobias Batton (serial entrepreneur, product manager, game designer). Then others joined around us.

Eventually we formed a small (but awesome) startup team and began working on a prototype and eventually an alpha. We debuted a closed beta preview at TechCrunch Disrupt last spring and received enthusiastic reviews. Now, today, we are releasing our public beta.

Read the full press release here.

I hope you like what we’ve created so far. But please note it is still a BETA. We are interested in your feedback and we already have a lot of feedback from our private beta. Here are some of the ideas we are working on for our next few releases:

• The Number One request we have received so far is to make it easier and faster for people to find events that would interest them. So for the remainder of the year one of our big priorities will be to add in more personalization and recommendations.
• We’re also working on new UI concepts, including some more ways to view the schedule of the Web.
• And we’re going to make it easier and faster for you to add events to Live Matrix — we’ll be launching improvements to our publisher tools section, as well more ways for people to suggest events for us to list.
• And we also plan to add new categories of events — for examples, Business, Technology, Games, and more.

So stay tuned! Live Matrix is just getting started. But this could be the start of something big.

ps. Here’s a screencast with a quick tour of Live Matrix

Live Matrix Demo from Doug Freeman on Vimeo.

If 2010 was the year of the Real-Time Web, then 2011 is going to be the year that it evolves into the Scheduled Web.

The Real-Time Web happens in the now: it is spontaneous, overwhelming, and disorganized. Things just happen unpredictably and nobody really knows what to expect or what will happen when.

The Real-Time Web is something of a misnomer, however, because usually it’s not real-time at all –  it’s after-the-fact. Most people find out about things that happened on the Real-Time Web after they happen, or, if they are lucky, when they happen. There is no way to know what is going to happen before it happens; there is no way to prepare or ensure that you will be online when something happens on the Real-Time Web. It’s entirely hit-or-miss.

If we are going to truly realize the Real-Time Web vision, then “time” needs to be the primary focus. So far, the Real-Time Web has mainly just been about simultaneity and speed – for example how quickly people on Twitter can respond to an event in the real world such as the Haiti Earthquake or the Oscars.

This obsession with the present is a sign of the times, but it is also a form of collective myopia — the Real-Time Web really doesn’t include the past or the future – it exists in a kind of perpetual now. To put the “time” into Real-Time, we need to  provide a way to see the past, present and the future Real-Time Web at once.  For example, we need a way to search and browse the past, present, and the future of a stream – what happened, what is happening, and what is scheduled to happen in the future. And this is where what I am calling The Scheduled Web comes in. It’s the next step for the Real-Time Web.

Defining the Scheduled Web

With the Scheduled Web things will start to make sense again. There will be a return of some semblance of order thanks to schedule metadata that enables people (and software) to find out about upcoming things on the Web that matter to them, before they happen, and to find out about past things that matter, after they happen.

The Scheduled Web is a Web that has a schedule, or many schedules, which exist in some commonly accessible, open format. These schedules should be searchable, linkable, shareable, interactive, collaborative, and discoverable. And they should be able to apply to anything — not just video, but any kind of content or activity online.

Why is this needed? Well consider this example. Imagine if there was no TV Guide on digital television. How would you navigate the constantly changing programming of more than 1000 digital TV channels without an interactive program guide (IPG)? It would be extremely difficult to find shows in a timely manner. According to clickstream data from television set-top boxes, about 10% of all time spent watching TV is spent in the IPG environment. And that is not even counting additional time-spent in on-demand guidance interfaces on DVRs. The point here is that guidance is key when you have lots of streams of content happening over time.

Now extend this same problem to the Web where there are literally millions of things happening every minute. These streams of content are not just limited to video. There are myriad types of real-time streams, everything from sales, auctions, and chats, to product launches, games, and audio, to streams of RSS feeds, Web pages appearing on Web sites, photos appearing on photo sites, software releases, announcements, etc.

Without some kind of guidance it is simply impossible to navigate the firehose of live online content streams on the Web efficiently. This firehose is too much to cope with in the present moment, let alone the past, or the future. This is what the Scheduled Web will solve.

By giving people a way to see into the past, present and future of the Real-Time Web, the Scheduled Web will enable the REAL Real-Time Web to be truly actualized. People will be able to know and plan in advance to actually be online when live events they care about take place.

Instead of missing that cool live Web concert or that auction for your favorite brand of shoes, simply because you didn’t know about it beforehand, you will be able to discover it in advance, RSVP, and get reminded before it starts — so you can be there and participate in the experience, right as it happens.

We are just beginning to see the emergence of the Scheduled Web. Two new examples of startups that are at work in the space are Clicker and Live Matrix.

• Clicker, a site that mainly provides on-demand video clips of past TV episodes, this week launched a schedule for live video streams on the Web.

• Live Matrix (my new startup), is soon to launch a schedule for all types of online events, not just video streams.

Some people have compared Live Matrix to Clicker, however this is not a wholly accurate comparison. We have very different, although  intersecting, goals.

While Clicker is an interesting play to compete with TV Guide and companies like Hulu, Live Matrix is creating a broader index of all the events taking place across the Scheduled Web, not just video/TV content events.

The insight behind Live Matrix is that there is much more to the Scheduled Web than video and TV content. The Web is not just about TV or video – it is about many different kinds of content.

Applying a TV metaphor to the Web is like trying to apply a print metaphor to tablet computing. While print has many positive qualities, tablet devices should not be limited just to text should they? Likewise, while the TV metaphor has advantages, it doesn’t make sense to limit the experience of time or scheduled content on the Web just to video.

With this in mind, while Live Matrix includes scheduled live video streams, we view video and TV type content as just one of many different types of scheduled Web content that matter.

For example, Live Matrix also includes online shopping events like sales and auctions, which comprise an enormous segment of the Scheduled Web. As an illustration eBay alone lists around 10 million scheduled auctions and sales each day! Live Matrix also includes scheduling metadata for many other kinds of content — online games, online chats, online audio, and more.

Live Matrix is building something quite a bit broader than current narrow conceptions of the Real-Time Web, or the narrow metaphor of TV on the Web. We are creating a way to navigate and search the full time dimension of the Web, we are building the schedule of the Web.

This will become a valuable, even essential, layer of metadata that just about every application, service and Internet surfer will make use of every day. Because after all, life happens in time and so does the Web. By adding metadata about time to the Web, Live Matrix will help make the Web – and particularly the Real-Time Web – easier to navigate.

Online vs. Offline Events

One of the key rules of Live Matrix is that, to be included in our schedule, an event must be consumable on-line. This means that it must be possible to access and participate in an event on an Internet-connected device.

Live Matrix is not a schedule of offline events or events that cannot be consumed or participated in using Internet-connected devices.

We made this rule because we believe that in the near-future almost everything interesting will, in fact, be consumable online, even if it has an offline component to it. We want to focus attention on those events which can be consumed on Internet-connected devices, so that if you have a connected device you can know that everything in Live Matrix can be accessed directly on your device. You don’t have to get in your car and drive to some physical venue, you don’t have to leave the Internet and go to some other device and network (like a TV and cable network).

Note the shift in emphasis here: We believe that the center of an increasing number of events is going to be online, and the offline world is going to increasingly become more peripheral.

For example, if a retail sale generates more revenues from online purchases than physical in-store purchases, the center of the sale is really on-line and the physical store becomes peripheral. Similarly, if a live concert has 30,000 audience members in a physical stadium but 10,000,000 people attending it online, the bulk of the concert is in fact online. This is already starting to happen.

For example, the recent Youtube concert featuring U2 had 10 million live streams – that’s up to 10 million live people in the audience at one time, making it possibly the largest online concert in history; it’s certainly a lot more people than any physical stadium could accommodate. Similarly, online venues like Second Life and World of Warcraft can accommodate thousands of players interacting in the same virtual spaces – not only do these spaces not even have a physical analogue (they exist only in virtual space), but there are no physical spaces that could accommodate such large games. These are examples of how online events may start to eclipse offline events.

I’m not saying this trend is good or bad; I’m simply stating a fact of our changing participatory culture. The world is going increasingly online and with this shift the center of our lives is going increasingly online, as well. It is this insight that gave my co-founder, Sanjay Reddy, and I, the inspiration to start Live Matrix, and to begin building what we hope will be the backbone of the Scheduled Web.

Today I am announcing that my company, Radar Networks, and its flagship product, Twine, have been acquired by Evri. TechCrunch broke the story here.

This acquisition consolidates two leading providers of semantic discovery and search. It is also the culmination of a long and challenging venture to pioneer the adoption of the consumer Semantic Web.

As the CEO and founder of Radar Networks and Twine.com, it is difficult to describe what it feels like to have reached this milestone during what has been a tumultuous period of global recession. I am very proud of my loyal and dedicated team and the incredible work and accomplishments that we have made together, and I am grateful for the unflagging support of our investors, and the huge community of Twine users and supporters.

Selling Twine.com was not something we had planned on doing at this time, but given the economy and the fact that Twine.com is a long-term project that will require significant ongoing investment and work to reach our goals, it is the best decision for the business and our shareholders.

While we received several offers for the company, and were in discussions about M&A with multiple industry leading companies in media, search and social software, we eventually selected Evri.

The Twine team is joining Evri to continue our work there. The Evri team has assured me that Twine.com’s data and users are safe and sound and will be transitioned into the Evri.com service over time, in a manner that protects privacy and data, and is minimally disruptive. I believe they will handle this with care and respect for the Twine community.

It is always an emotional experience to sell a company. Building Twine.com has been a long, intense, challenging, rewarding, and all-consuming effort. There were incredible high points and some very deep lows along the way. But most of all, it has been an adventure I will never forget. I was fortunate to help pioneer a major new technology — the Semantic Web — with an amazing team, including many good friends. Bringing something as big, as ambitious, and as risky as Twine.com to market was exhilarating.

Twine has been one of the great learning experiences of my life. I am profoundly grateful to everyone I’ve worked with, and especially to those who supported us financially and personally with their moral support, ideas and advocacy.

I am also grateful to unsung heroes behind the project — the families of all of us who worked on it, who never failed to be supportive as we worked days, nights, weekends and vacations to bring Twine to market.

What I’m Doing Next

I will advise Evri through the transition, but will not be working full-time there. Instead, I will be turning my primary focus to several new projects, including some exciting new ventures:

• Live Matrix, a new venture focusing on making the live Web more navigable. Live Matrix is led by Sanjay Reddy (CEO of Live Matrix; formerly SVP of Corp Dev for Gemstar TV Guide). Live Matrix is going to give the Web a new dimension: time. More news about this soon.
• Klout, the leading provider of social analytics about influencers on Twitter and Facebook (which I was the first angel investor in, and which I now advise). Klout is a really hot  company and it’s growing fast.
• I’m experimenting with a new way to grow ventures. It’s part incubator, part fund, part production company. I call it a Venture Production Studio. Through this initiative my partners and I are planning to produce a number of original startups, and selected outside startups as well. There is a huge gap in the early-stage arena, and to fill this we need to modify the economics and model of early stage venture investing.
• I’m looking forward to working more on my non-profit interests, particularly those related to supporting democracy and human rights around the world, and one of my particular interests, Tibetan cultural preservation.
• And last but not least, I’m getting married later this month, which may turn out to be my best project of all.

If you want to keep up with what I am thinking about and working on, you should follow me on Twitter at @novaspivack, and also keep up with my blog here at novaspivack.com and my mailing list (accessible in the upper right hand corner of this page).

The Story Behind the Story

In making this transition, it seems appropriate to tell the Twine.com story. This will provide some insight into how we got here, including some of our triumphs, and our mistakes, and some of the difficulties we faced along the way. Hopefully this will shed some light on the story behind the story, and may even be useful to other entrepreneurs out there in what is perhaps one of the most difficult venture capital and startup environments in history.

(Note: You may also be interested in viewing this presentation, “A Yarn About Twine” which covers the full history of the project with lots of pictures of various iterations of our work from the early semantic desktop app to Twine, to T2.)

The Early Years of the Project

The ideas that led to Twine were born in the 1990′s from my work as a co-founder of EarthWeb (which today continues as Dice.com), where among many things we prototyped a number of new knowledge-sharing and social networking tools, along with our primary work developing large Web portals and communities for customers, and eventually our own communities for IT professionals. My time with EarthWeb really helped me to understand that challenges and potential of sharing and growing knowledge socially on the Web. I became passionately interested in finding new ways to network people’s minds together, to solve information overload, and to enable the evolution of a future “global brain.”

After EarthWeb’s IPO I worked with SRI and Sarnoff to build their business incubator, nVention, and then eventually started my own incubator, Lucid Ventures, through which I co-founded Radar Networks with Kristin Thorisson, from the MIT Media Lab, and Jim Wissner (the continuing Chief Architect of Twine) in 2003. Our first implementation was a peer-to-peer Java-based knowledge sharing app called “Personal Radar.”

Personal Radar was a very cool app — it organized all the information on the desktop in a single semantic information space that was like an “iTunes for information” and then made it easy to share and annotate knowledge with others in a collaborative manner. There were some similarities to apps like Ray Ozzie’s Groove and the MIT Haystack project, but Personal Radar was built for consumers, entirely with Java, RDF, OWL and the standards of the emerging Semantic Web. You can see some screenshots pictures of this early work in this slideshow, here.

But due to the collapse of the first Internet bubble there was simply no venture funding available at the time and so instead, we ended up working as subcontractors on the DARPA CALO project at SRI. This kept our research alive through the downturn and also introduced us to a true Who’s Who of AI and Semantic Web gurus who worked on the CALO project. We eventually helped SRI build OpenIRIS, a personal semantic desktop application, which had many similarities to Personal Radar. All of our work for CALO was open-sourced under the LGPL license.

Becoming a Venture-Funded Company

Deborah L. McGuinness, who was one of the co-designers of the OWL language (the Web Ontology Language, one of the foundations of the Semantic Web standards at the W3C), became one of our science advisers and kindly introduced us to Paul Allen, who invited us to present our work to his team at Vulcan Capital. The rest is history. Paul Allen and Ron Conway led an angel round to seed-fund us and we moved out of consulting to DARPA and began work on developing our own products and services.

Our long-term plan was to create a major online portal powered by the Semantic Web that would provide a new generation of Web-scale semantic search and discovery features to consumers. But for this to happen, first we had to build our own Web-scale commercial semantic applications platform, because there was no platform available at that time that could meet the requirements we had. In the process of building our platform numerous technical challenges had to be overcome.

At the time (the early 2000′s) there were few development tools in existence for creating ontologies or semantic applications, and in addition there were no commercial-quality databases capable of delivering high-performance Web-scale storage and retrieval of RDF triples. So we had to develop our own development tools, our own semantic applications framework, and our own federated high-performance semantic datastore.

This turned out to be a nearly endless amount of work. However we were fortunate to have Jim Wissner as our lead technical architect and chief scientist. Under his guidance we went through several iterations and numerous technical breakthroughs, eventually developing the most powerful and developer-friendly semantic applications platform in the world. This led to the  development of a portfolio of intellectual property that provides fundamental DNA for the Semantic Web.

During this process we raised a Series A round led by Vulcan Capital and Leapfrog Ventures, and our team was joined by interface designer and product management expert, Chris Jones (now leading strategy at HotStudio, a boutique design and user-experience firm in San Francisco). Under Chris’ guidance we developed Twine.com, our first application built on our semantic platform.

The mission of Twine.com was to help people keep up with their interests more efficiently, using the Semantic Web. The basic idea was that you could add content to Twine (most commonly by bookmarking it into the site, but also by authoring directly into it), and then Twine would use natural language processing and analysis, statistical methods, and graph and social network analysis, to automatically store, organize, link and semantically tag the content into various topical areas.

These topics could easily be followed by other users who wanted to keep up with specific types of content or interests. So basically you could author or add stuff to Twine and it would then do the work of making sense of it, organizing it, and helping you share it with others who were interested. The data was stored semantically and connected to ontologies, so that it could then be searched and reused in new ways.

With the help of Lew Tucker, Sonja Erickson and Candice Nobles, as well as an amazing team of engineers, product managers, systems admins and designers, Twine was announced at the Web 2.0 Summit in October of 2007 and went into full public beta in Q1 of 2008. Twine was well-received by the press and early-adopter users.

Soon after our initial beta launch we raised a Series B round, led by Vulcan Capital and Velocity Interactive Group (now named Fuse Capital), as well as DFJ. This gave us the capital to begin to grow Twine.com rapidly to become the major online destination we envisioned.

In the course of this work we made a number of additional technical breakthroughs, resulting in more than 20 patent filings in total, including several fundamental patents related to semantic data management, semantic portals, semantic social networking, semantic recommendations, semantic advertising, and semantic search.

Four of those patents have been granted so far and the rest are still pending — and perhaps the most interesting of these patents are related to our most recent work on “T2″ and are not yet visible.

At the time of beta launch and for almost six months after, Twine was still very much a work in progress. Fortunately our users and the press were fairly forgiving as we worked through evolving the GUI and feature set from what was initially just slightly better than an alpha site to the highly refined and graphical UI we have today.

During these early days of Twine.com we were fortunate to have a devoted user-base and this became a thriving community of power-users who really helped us to refine the product and develop great content within it.

Rapid Growth, and Scaling Challenges

As Twine grew the community went through many changes and some growing pains, and eventually crossed the chasm to a more mainstream user-base. Within less than a year from launch the site grew to around 3 million monthly visitors, 300,000 registered users, 25,000 “twines” about various interests, and almost 5 million pieces of user-contributed content. It was on its way to becoming the largest semantic web on the Web.

By all accounts Twine was looking like a potential “hit.” During this period the company staff increased to more than 40 people (inclusive of contractors and offshore teams) and our monthly burn rate increased to aggressive levels of spending to keep up with growth.

Despite this growth and spending we still could not keep up with demand for new features and at times we experienced major scaling and performance challenges. We had always planned for several more iterations of our backend architecture to facilitate scaling the system. But now we could see the writing on the wall — we had to begin to develop a more powerful, more scalable backend for Twine, much sooner than we had expected we would need to.

This required us to increase our engineering spending further in order to simultaneously support the live version of Twine and its very substantial backend, and run a parallel development team working on the next generation of the backend and the next version of Twine on top of it. Running multiple development teams instead of one was a challenging and costly endeavor. The engineering team was stretched thin and we were all putting in 12 to 15 hour days every day.

Breakthrough to “T2″

We began to work in earnest on a new iteration of our back-end architecture and application framework — one that could scale fast enough to keep up with our unexpectedly fast growth rate and the increasing demands on our servers that this was causing.

This initiative yielded unexpected fruit. Not only did we solve our scaling problems, but we were able to do so to such a degree that entirely new possibilities were opened up to us — ones that had previously been out of reach for purely technical reasons. In particular, semantic search.

Semantic search had always been a long-term goal of ours, however, in the first version of Twine (the one that is currently online) search was our weakest feature area, due to the challenge of scaling a semantic datastore to handle hundreds of billions of triples. But our user-studies revealed that it was in fact the feature our users wanted us to develop the most – search slowly became the dominant paradigm within Twine, especially when the content in our system reached critical mass.

Our new architecture initiative solved the semantic search problem to such a degree that we realized that not only could we scale Twine.com, we could scale it to eventually become a semantic search engine for the entire Web.

Instead of relying on users to crowdsource only a subset of the best content into our index, we could crawl large portions of the Web automatically and ingest millions and millions of Web pages, process them, and make them semantically searchable — using a true W3C Semantic Web compliant backend. (Note: Why did we even attempt to do this? We believed strongly in supporting open-standards for the Semantic Web, despite the fact that they posed major technical challenges and required tools that did not exist yet, because they promised to enable semantic application and data interoperability, one of the main potential benefits of the Semantic Web).

Based on our newfound ability to do Web-scale semantic search, we began planning the next version of Twine — Twine 2.0 (“T2″), with the help of Bob Morgan, Mark Erickson, Sasi Reddy, and a team of great designers.

The new T2 plan would merge new faceted semantic search features with the existing social, personalization and knowledge management features of Twine 1.0. It would be the best of both worlds: semantic search + social search. We began working intensively on developing T2, along with a new hosted developer tools that would make it easy for any webmaster to easily add their site into our semantic index. We were certain that with T2 we had finally “cracked the code” to the Semantic Web — we had a product plan and a strategy that could really bring the Semantic Web to everyone on the Web. It elegantly solved the key challenges to adoption and on a technical level, using SOLR instead of a giant triplestore, we were able to scale to unprecedented levels. It was an exciting plan and everyone on the team was confident in the direction.

To see screenshots that demo T2 and our hosted development tools click here.

The Global Recession

Our growth was fast, and so was our spending, but at the time this seemed logical because the future looked bright and we were in a race to keep ahead of our own curve. We were quickly nearing a point where we would soon need to raise another round of funding to sustain our pace, but we were confident that with our growth trends steadily increasing and our exciting plans for T2, the necessary funding would be forthcoming at favorable valuations.

We were wrong.

The global economy crashed unexpectedly, throwing a major curveball in our path. We had not planned on that happening and it certainly was inconvenient to say the least.

The recession not only hit Wall Street, it hit Silicon Valley. Venture capital funding dried up almost overnight. VC funds sent alarming letters to their portfolio companies warning of dire financial turmoil ahead. Many startups were forced to close their doors, while others made drastic sudden layoffs for better or for worse. We too made spending cuts, but we were limited in our ability to slash expenses until the new T2 platform could be completed. Once that was done, we would be able to move Twine to a much more scalable and less costly architecture, and we would no longer need parallel development teams. But until that happened, we still had to maintain a sizeable infrastructure and engineering effort.

As the recession dragged on, and the clock kept ticking down, the urgency of raising a C round increased, and finally we were faced with a painful decision. We had to drastically reduce our spending in order to wait out the recession and live to raise more funding in the future.

Unfortunately, the only way to accomplish such a drastic reduction in spending was to lay off almost 30% of our staff and cut our monthly spending by almost 40%. But by doing that we could not possibly continue to work on as many fronts as we had been doing. The result was that we had to stop most work on Twine 1.0 (the version that was currently online) and focus all our remaining development cycles and spending on the team needed to continue our work on T2.

This was extremely painful for me as the CEO, and for everyone on our team. But it was necessary for the survival of the business and it did buy us valuable time. However, it also slowed us down tremendously. The irony of making this decision was that it reduced our burn-rate but slowed us down, reduced productivity, and cost us time to such a degree that in the end it may have cost us the same amount of money anyway.

While much of our traffic had been organic and direct, we also had a number of marketing partnerships and PR initiatives that we had to terminate. In addition, as part of this layoff we lost our amazing and talented marketing team, as well as half our product management team, our entire design team, our entire marketing and PR budget, and much of our support and community management team. This made it difficult to continue to promote the site, launch new features, fix bugs, or to support our existing online community. And as a result the service began to decline and usage declined along with it.

To make matters worse, at around the same time as we were making these drastic cuts, Google decided to de-index Twine. To this day we still are not sure why they decided to do this – it could have been that Google suddenly decided we were a competitive search engine, or it could be that their algorithm changed, or it could be that there was some error in our HTML markup that may have caused an indexing problem. We had literally millions of pages of topical user-generated content – but all of a sudden we saw drastic reductions in the number of pages being indexed, and in the ranking of those pages. This caused a very significant drop in organic traffic. With what little team I had remaining we spent time petitioning Google and trying to get reinstated. But we never managed to return to our former levels of index prominence.

Eventually, with all these obstacles, and the fact that we had to focus our remaining budget on T2, we put Twine.com on auto-pilot and let the traffic fall off, believing that we would have the opportunity to win it back once we launched next versipn. While painful to watch, this reduction in traffic and user activity at least had the benefit of reducing the pressure on the engineering team to scale the system and support it under load, giving us time to focus all our energy on getting T2 finished and on raising more funds.

But the recession dragged on and on and on, without end. VC’s remained extremely conservative and risk-averse. Meanwhile, we focused our internal work on growing a large semantic index of the Web in T2, vertical by vertical, starting with food, then games, and then many other topics (technology, health, sports, etc.). We were quite confident that if we could bring T2 to market it would be a turning point for Web search, and funding would follow.

Meanwhile we met with VC’s in earnest. But nobody was able to invest in anything due to the recession. Furthermore we were a pre-revenue company working on a risky advanced technology and VC partnerships were far too terrified by the recession to make such a bet. We encountered the dreaded “wait and see” response.

The only way we could get the funding we needed to continue was to launch T2, grow it, and generate revenues from it, but the only way we could reach those milestones was to launch T2 in the first place: a classic catch-22 situation.

We took comfort in the fact that we were not alone in this predicament. Almost every tech company at our stage was facing similar funding challenges. However, we were determined to find a solution despite the obstacles in our path.

Selling the Business

Had the recession not happened, I believe we would have raised a strong C round based on the momentum of the product and our technical achievements. Unfortunately, we, like many other early-stage technology ventures, found ourselves in the worst capital crunch in decades.

We eventually came to the conclusion that there was no viable path for the company but to use the runway we had left to sell to another entity that was more able to fund the ongoing development and marketing necessary to monetize T2.

While selling the company had always been a desirable exit strategy, we had hoped to do it after the launch and growth of T2. However, we could not afford to wait any longer. With some short-term bridge funding from our existing investors, we worked with Growth Point Technology Partners to sell the company.

We met with a number of the leading Internet and media companies and received numerous offers. In the end, the best and most strategically compatible offer came from Evri, one of our sibling companies in Vulcan Capital’s portfolio. While we had the option to sell to larger and more established companies with very compelling offers, it was simply the best option to join Evri.

And so we find ourselves at the present day. We got the best deal possible for our shareholders given the circumstances. Twine.com, my team, our users and their data are safe and sound. As an entrepreneur and CEO it is, as one advisor put it, of the utmost importance to always keep the company moving forward. I feel that I did manage to achieve this under extremely difficult economic circumstances. And for that I am grateful.

Outlook for the Semantic Web

I’ve been one of the most outspoken advocates of the Semantic Web during my tenure at Twine. So what about my outlook for the Semantic Web now that Twine is being sold and I’m starting to do other things? Do I still believe in the promise of the Semantic Web vision? Where is it going? These are questions I expect to be asked, so I will attempt to answer them here.

I continue to believe in the promise of semantic technologies, and in particular the approach of the W3C semantic web standards (RDF, OWL, SPARQL). That said, having tried to bring them to market as hard as anyone ever has, I can truly say they present significant challenges both to developers and to end-users. These challenges all stem from one underlying problem: Data storage.

Existing SQL databases are not optimal for large-scale, high-performance semantic data storage and retrieval. Yet triplestores are still not ready for prime-time. New graph databases and column stores show a lot of promise, but they are still only beginning to emerge. This situation makes it incredibly difficult to bring Web-scale semantic applications to market cost-effectively.

Enterprise semantic applications are much more feasible today however — because existing and emerging databases and semantic storage solutions do scale to enterprise levels. But for consumer-grade, enormous, Web services, there are still challenges. This is single greatest technical obstacle that Twine faced and it cost us a large amount of our venture funding to surmount. Finally we did find a solution with our T2 architecture, but it is still not a general solution for all types of applications.

I have recently seen some new graph data storage products that may provide the levels of scale and performance needed, but pricing has not been determined yet. In short, storage and retrieval of semantic graph datasets is a big unsolved challenge that is holding back the entire industry. We need federated database systems that can handle hundreds of billions to trillions of triples under high load conditions, in the cloud, on commodity hardware and open source software. Only then will it be affordable to make semantic applications and services at Web-scale.

I believe that semantic metadata is essential for the growth and evolution of the Web. It is one of the only ways we can hope to dig out from the increasing problem of information overload. It is one of the only ways to make search, discovery, and collaboration smart enough to really be significantly better than it is today.

But the notion that everyone will learn and adopt standards for creating this metadata themselves is flawed in my opinion. They won’t. Instead, we must focus on solutions (like Twine and Evri) that make this metadata automatically by analyzing content semantically. I believe this is the most practical approach to bringing the value of semantic search and discovery to consumers, as well as Webmasters and content providers around the Web.

The major search engines are all working on various forms of semantic search, but to my knowledge none of them are fully supporting the W3C standards for the Semantic Web. In some cases this is because they are attempting to co-opt the standards for their own competitive advantage, and in other cases it is because it is simply easier not to use them. But in taking the easier path, they are giving up the long-term potential gains of a truly open and interoperable semantic ecosystem.

I do believe that whoever enables this open semantic ecosystem first will win in the end — because it will have greater and faster network effects than any closed competing system. That is the promise and beauty of open standards: everyone can feel safe using them since no single commercial interest controls them. At least that’s the vision I see for the Semantic Web.

As far as where the Semantic Web will add the most value in years to come, I think we will see it appear in some new areas. First and foremost is e-commerce, an area that is ripe with structured data that needs to be normalized, integrated and made more searchable. This is perhaps the most potentially profitable and immediately useful application of semantic technologies. It’s also one where there has been very little innovation. But imagine if eBay or Amazon or Salesforce.com provided open-standards-compliant semantic metadata and semantic search across all their data.

Another important opportunity is search and SEO — these are the areas that Twine’s T2 project focused on, by enabling webmasters to easily and semi-automatically add semantic descriptions of their content into search indexes, without forcing them to learn RDF and OWL and do it manually. This would create a better SEO ecosystem and would be beneficial not only to content providers and search engines, but also to advertisers. This is the approach that I believe the major search engines should take.

Another area where semantics could add a lot of value is social media — by providing semantic descriptions of user profiles and user profile data, as well as social relationships on the Web, it would be possible to integrate and search across all social networks in a unified manner.

Finally, another area where semantics will be beneficial is to enable easier integration of datasets and applications around the Web — currently every database is a separate island, but by using the Semantic Web appropriately data can be freed from databases and easily reused, remixed and repurposed by other applications. I look forward to the promise of a truly open data layer on the Web, when the Web becomes essentially one big open database that all applications can use.

Lessons Learned and Advice for Startups

While the outcome for Twine was decent under the circumstances, and was certainly far better than the alternative of simply running out of money, I do wonder how it could have been different. I ask myself what I learned and what I would do differently if I had the chance or could go back in time.

I think the most important lessons I learned, and the advice that I would give to other entrepreneurs can be summarized with a few key points:

1. Raise as little venture capital as possible. Raise less than you need, not more than you need. Don’t raise extra capital just because it is available. Later on it will make it harder to raise further capital when you really need it. If you can avoid raising venture capital at all, do so. It comes with many strings attached. Angel funding is far preferable. But best of all, self-fund from revenues as early as you can, if possible. If you must raise venture capital, raise as little as you can get by on — even if they offer you more. But make sure you have at least enough to reach your next funding round — and assume that it will take twice as long to close as you think. It is no easy task to get a startup funded and launched in this economy — the odds are not in your favor — so play defense, not offense, until conditions improve (years from now).
2. Build for lower exits. Design your business model and capital strategy so that you can deliver a good ROI to your investors at an exit under $30mm. Exit prices are going lower, not higher. There is less competition and fewer buyers and they know it’s a buyer’s market. So make sure your capital strategy gives the option to sell in lower price ranges. If you raise too much you create a situation where you either have to sell at a loss, or raise even more funding which only makes the exit goal that much harder to reach.
3. Spend less. Spend less than you want to, less than you need to, and less than you can. When you are flush with capital it is tempting to spend it and grow aggressively, but don’t. Assume the market will crash — downturns are more frequent and last longer than they used to. Expect that. Plan on it. And make sure you keep enough capital in reserve to spend 9 to 12 months raising your next round, because that is how long it takes in this economy to get a round done.
4. Don’t rely on user-traction to raise funding. You cannot assume that user traction is enough to get your next round done. Even millions of users and exponential growth are not enough. VC’s and their investment committees want to see revenues, and particularly at least breakeven revenues. A large service that isn’t bringing in revenues yet is not a business, it’s an experiment. Perhaps it’s one that someone will buy, but if you can’t find a buyer then what? Don’t assume that VC’s will fund it. They won’t. Venture capital investing has changed dramatically — early stage and late stage deals are the only deals that are getting real funding. Mid-stage companies are simply left to die, unless they are profitable or will soon be profitable.
5. Don’t be afraid to downsize when you have to. It sucks to fire people, but it’s sometimes simply necessary. One of the worst mistakes is to not fire people who should be fired, or to not do layoffs when the business needs require it. You lose credibility as a leader if you don’t act decisively. Often friendships and personal loyalties prevent or delay leaders from firing people that really should be fired. While friendship and loyalty are noble they unfortunately are not always the best thing for the business. It’s better for everyone to take their medicine sooner rather than later. Your team knows who should be fired. Your team knows when layoffs are needed. Ask them. Then do it. If you don’t feel comfortable firing people, or you can’t do it, or you don’t do it when you need to, don’t be the CEO.
6. Develop cheaply, but still pay market salaries. Use offshore development resources, or locate your engineering team outside of the main “tech hub” cities. It is simply too expensive to compete with large public and private tech companies to pay top dollar for engineering talent in places like San Francisco and Silicon Valley.  The cost of top-level engineers is too high in major cities to be affordable and the competition to hire and retain them is intense. If you can get engineers to work for free or for half price then perhaps you can do it, but I believe you get what you pay for. So rather thank skimp on salaries, pay people market salaries, but do it where market salaries are more affordable.
7. Only innovate on one frontier at a time. For example, either innovate by making a new platform, or a new application, or a new business model. Don’t do all of these at once, it’s just too hard. If you want to make a new platform, just focus on that, don’t try to make an application too. If you want to make a new application, use an existing platform rather than also building a platform for it. If you want to make a new business model, use an existing application and platform — they can be ones you have built in the past, but don’t attempt to do it all at once. If you must do all three, do them sequentially, and make sure you can hit cash flow breakeven at each stage, with each one. Otherwise you’re at risk in this economy.

I hope that this advice is of some use to entrepreneurs (and VC’s) who are reading this. I’ve personally made all these mistakes myself, so I am speaking from experience. Hopefully I can spare you the trouble of having to learn these lessons the hard way.

What we did Well

I’ve spent considerable time in this article focusing on what didn’t go according to plan, and the mistakes we’ve learned from. But it’s also important to point out what we did right. I’m proud of the fact that Twine accomplished many milestones, including:

• Pioneering the Semantic Web and leading the charge to make it a mainstream topic of conversation.
• Creating the most powerful, developer friendly, platform for the Semantic Web.
• Successfully completing our work on CALO, the largest Semantic Web project in the US.
• Launching the first mainstream consumer application of Semantic Web.
• Having a very successful launch, covered by hundreds of articles.
• Gaining users extremely rapidly — faster than Twitter did in it’s early years.
• Hiring and retaining an incredible team of industry veterans.
• Raising nearly $24mm of venture capital over 2 rounds, because our plan was so promising.
• Developing more than 20 patents, several of which are fundamentally important for the Semantic Web field.
• Surviving two major economic bubbles and the downturns that followed.
• Innovating and most of all, adapting to change rapidly.
• Breaking through to T2 — a truly awesome technological innovation for Web-scale semantic search.
• Selling the company in one of the most difficult economic environments in history.

I am proud of what we accomplished with Twine. It’s been “a long strange trip” but one that has been full of excitement and accomplishments to remember.

Conclusions

If you’ve actually read this far, thank you. This is a big article, but after all, Twine is a big project – One that lasted nearly 5 years (or 9 years if you include our original research phase). I’m still bullish on the Semantic Web, and genuinely very enthusiastic about what Evri will do with Twine.com going forward.

Again I want to thank the hundreds of people who have helped make Twine possible over the years – but in particular the members of our technical and management team who went far beyond the call of duty to get us to the deal we have reached with Evri.

While this is certainly the end of an era, I believe that this story has only just begun. The first chapters are complete and now we are moving into a new era. Much work remains to be done and there are certainly still challenges and unknowns, but progress continues and the Semantic Web is here to stay.

The human brain is like an archaeological record. Different layers and functional areas have evolved outwards over time. And now a new layer is evolving. I propose we call this new layer of the brain “the metacortex.” (Note: Metacortex also happens to be the company that Neo worked for in the movie, The Matrix)

The metacortex is the Web — our growing global network of information, people, sensors, and computing devices.

The Web is literally a new layer of the human brain that transcends any individual brain. It is a global brain that connects all our brains together. It is intelligent. It is perhaps humanity’s greatest invention.  It collectively senses, reacts, interprets, learns, thinks, and acts in ways that we as individuals can barely comprehend or predict, and this activity comprises an emerging global mind.

Paul Buchheit (creator of Gmail and Friendfeed) calls this “the social brain” — with emphasis on the social networks and collective social interactions that are taking place. I think that while the metacortex includes the social Web, it transcends it — it’s collective knowledge and cognition include all of the activity taking place on the Internet.

Does the metacortex mirror the structure and process of the neocortex? What can we learn about the neocortex from the metacortex and vice versa? What are the functional areas or lobes of the metacortex? I look forward to your comments.

Sneak Preview of Siri – The Virtual Assistant that will Make Everyone Love the iPhone, Part 2: The Technical Stuff

In Part-One of this article on TechCrunch, I covered the emerging paradigm of Virtual Assistants and explored a first look at a new product in this category called Siri. In this article, Part-Two, I interview Tom Gruber, CTO of Siri, about the history, key ideas, and technical foundations of the product:

Nova Spivack: Can you give me a more precise definition of a Virtual Assistant?

Tom Gruber: A virtual personal assistant is a software system that

• Helps the user find or do something (focus on tasks, rather than information)
• Understands the user’s intent (interpreting language) and context (location, schedule, history)
• Works on the user’s behalf, orchestrating multiple services and information sources to help complete the task

In other words, an assistant helps me do things by understanding me and working for me. This may seem quite general, but it is a fundamental shift from the way the Internet works today. Portals, search engines, and web sites are helpful but they don’t do things for me – I have to use them as tools to do something, and I have to adapt to their ways of taking input.

Nova Spivack: Siri is hoping to kick-start the revival of the Virtual Assistant category, for the Web. This is an idea which has a rich history. What are some of the past examples that have influenced your thinking?

Tom Gruber: The idea of interacting with a computer via a conversational interface with an assistant has excited the imagination for some time. Apple’s famous Knowledge Navigator video offered a compelling vision, in which a talking head agent helped a professional deal with schedules and access information on the net. The late Michael Dertouzos, head of MIT’s Computer Science Lab, wrote convincingly about the assistant metaphor as the natural way to interact with computers in his book “The Unfinished Revolution: Human-Centered Computers and What They Can Do For Us”. These accounts of the future say that you should be able to talk to your computer in your own words, saying what you want to do, with the computer talking back to ask clarifying questions and explain results. These are hallmarks of the Siri assistant. Some of the elements of these visions are beyond what Siri does, such as general reasoning about science in the Knowledge Navigator. Or self-awareness a la Singularity. But Siri is the real thing, using real AI technology, just made very practical on a small set of domains. The breakthrough is to bring this vision to a mainstream market, taking maximum advantage of the mobile context and internet service ecosystems.

Nova Spivack: Tell me about the CALO project, that Siri spun out from. (Disclosure: my company, Radar Networks, consulted to SRI in the early days on the CALO project, to provide assistance with Semantic Web development)

Tom Gruber: Siri has its roots in the DARPA CALO project (“Cognitive Agent that Learns and Organizes”) which was led by SRI. The goal of CALO was to develop AI technologies (dialog and natural language understanding,s understanding, machine learning, evidential and probabilistic reasoning, ontology and knowledge representation, planning, reasoning, service delegation) all integrated into a virtual assistant that helps people do things. It pushed the limits on machine learning and speech, and also showed the technical feasibility of a task-focused virtual assistant that uses knowledge of user context and multiple sources to help solve problems.

Siri is integrating, commercializing, scaling, and applying these technologies to a consumer-focused virtual assistant. Siri was under development for several years during and after the CALO project at SRI. It was designed as an independent architecture, tightly integrating the best ideas from CALO but free of the constraints of a national distributed research project. The Siri.com team has been evolving and hardening the technology since January 2008.

Nova Spivack: What are primary aspects of Siri that you would say are “novel”?

Tom Gruber: The demands of the consumer internet focus — instant usability and robust interaction with the evolving web — has driven us to come up with some new innovations:

• A conversational interface that combines the best of speech and semantic language understanding with an interactive dialog that helps guide people toward saying what they want to do and getting it done. The conversational interface allows for much more interactivity that one-shot search style interfaces, which aids usability and improves intent understanding. For example, if Siri didn’t quite hear what you said, or isn’t sure what you meant, it can ask for clarifying information. For example, it can prompt on ambiguity: did you mean pizza restaurants in Chicago or Chicago-style pizza places near you? It can also make reasonable guesses based on context. Walking around with the phone at lunchtime, if the speech interpretation comes back with something garbled about food you probably meant “places to eat near my current location”. If this assumption isn’t right, it is easy to correct in a conversation.
• Semantic auto-complete – a combination of the familiar “autocomplete” interface of search boxes with a semantic and linguistic model of what might be worth saying. The so-called “semantic completion” makes it possible to rapidly state complex requests (Italian restaurants in the SOMA neighborhood of San Francisco that have tables available tonight) with just a few clicks. It’s sort of like the power of faceted search a la Kayak, but packaged in a clever command line style interface that works in small form factor and low bandwidth environments.
• Service delegation – Siri is particularly deep in technology for operationalizing a user’s intent into computational form, dispatching to multiple, heterogeneous services, gathering and integrating results, and presenting them back to the user as a set of solutions to their request. In a restaurant selection task, for instance, Siri combines information from many different sources (local business directories, geospatial databases, restaurant guides, restaurant review sources, online reservation services, and the user’s own favorites) to show a set of candidates that meet the intent expressed in the user’s natural language request.

Nova Spivack: Why do you think Siri will succeed when other AI-inspired projects have failed to meet expectations?

Tom Gruber: In general my answer is that Siri is more focused. We can break this down into three areas of focus:

• Task focus. Siri is very focused on a bounded set of specific human tasks, like finding something to do, going out with friends, and getting around town. This task focus allows it to have a very rich model of its domain of competence, which makes everything more tractable from language understanding to reasoning to service invocation and results presentation
• Structured data focus. The kinds of tasks that Siri is particularly good at involve semistructured data, usually on tasks involving multiple criteria and drawing from multiple sources. For example, to help find a place to eat, user preferences for cuisine, price range, location, or even specific food items come into play. Combining results from multiple sources requires reasoning about domain entity identity and the relative capabilities of different information providers. These are hard problems of semantic information processing and integration that are difficult but feasible today using the latest AI technologies.
• Architecture focus. Siri is built from deep experience in integrating multiple advanced technologies into a platform designed expressly for virtual assistants. Siri co-founder Adam Cheyer was chief architect of the CALO project, and has applied a career of experience to design the platform of the Siri product. Leading the CALO project taught him a lot about what works and doesn’t when applying AI to build a virtual assistant. Adam and I also have rather unique experience in combining AI with intelligent interfaces and web-scale knowledge integration. The result is a “pure play” dedicated architecture for virtual assistants, integrating all the components of intent understanding, service delegation, and dialog flow management. We have avoided the need to solve general AI problems by concentrating on only what is needed for a virtual assistant, and have chosen to begin with a finite set of vertical domains serving mobile use cases.

Nova Spivack: Why did you design Siri primarily for mobile devices, rather than Web browsers in general?

Tom Gruber: Rather than trying to be like a search engine to all the world’s information, Siri is going after mobile use cases where deep models of context (place, time, personal history) and limited form factors magnify the power of an intelligent interface. The smaller the form factor, the more mobile the context, the more limited the bandwidth : the more it is important that the interface make intelligent use of the user’s attention and the resources at hand. In other words, “smaller needs to be smarter.” And the benefits of being offered just the right level of detail or being prompted with just the right questions can make the difference between task completion or failure. When you are on the go, you just don’t have time to wade through pages of links and disjoint interfaces, many of which are not suitable to mobile at all.

Nova Spivack: What language and platform is Siri written in?

Tom Gruber: Java, Javascript, and Objective C (for the iPhone)

Nova Spivack: What about the Semantic Web? Is Siri built with Semantic Web open-standards such as RDF and OWL, Sparql?

Tom Gruber: No, we connect to partners on the web using structured APIs, some of which do use the Semantic Web standards. A site that exposes RDF usually has an API that is easy to deal with, which makes our life easier. For instance, we use geonames.org as one of our geospatial information sources. It is a full-on Semantic Web endpoint, and that makes it easy to deal with. The more the API declares its data model, the more automated we can make our coupling to it.

Nova Spivack: Siri seems smart, at least about the kinds of tasks it was designed for. How is the knowledge represented in Siri – is it an ontology or something else?

Tom Gruber: Siri’s knowledge is represented in a unified modeling system that combines ontologies, inference networks, pattern matching agents, dictionaries, and dialog models. As much as possible we represent things declaratively (i.e., as data in models, not lines of code). This is a tried and true best practice for complex AI systems. This makes the whole system more robust and scalable, and the development process more agile. It also helps with reasoning and learning, since Siri can look at what it knows and think about similarities and generalizations at a semantic level.

Nova Spivack: Will Siri be part of the Semantic Web, or at least the open linked data Web (by making open API’s, sharing of linked data, RDF, available, etc.)?

Tom Gruber: Siri isn’t a source of data, so it doesn’t expose data using Semantic Web standards. In the Semantic Web ecosystem, it is doing something like the vision of a semantic desktop – an intelligent interface that knows about user needs and sources of information to meet those needs, and intermediates. The original Semantic Web article in Scientific American included use cases that an assistant would do (check calendars, look for things based on multiple structured criteria, route planning, etc.). The Semantic Web vision focused on exposing the structured data, but it assumes APIs that can do transactions on the data. For example, if a virtual assistant wants to schedule a dinner it needs more than the information about the free/busy schedules of participants, it needs API access to their calendars with appropriate credentials, ways of communicating with the participants via APIs to their email/sms/phone, and so forth. Siri is building on the ecosystem of APIs, which are better if they declare the meaning of the data in and out via ontologies. That is the original purpose of ontologies-as-specification that I promoted in the 1990s – to help specify how to interact with these agents via knowledge-level APIs.

Siri does, however, benefit greatly from standards for talking about space and time, identity (of people, places, and things), and authentication. As I called for in my Semantic Web talk in 2007, there is no reason we should be string matching on city names, business names, user names, etc.

All players near the user in the ecommerce value chain get better when the information that the users need can be unambiguously identified, compared, and combined. Legitimate service providers on the supply end of the value chain also benefit, because structured data is harder to scam than text. So if some service provider offers a multi-criteria decision making service, say, to help make a product purchase in some domain, it is much easier to do fraud detection when the product instances, features, prices, and transaction availability information are all structured data.

Nova Spivack: Siri appears to be able to handle requests in natural language. How good is the natural language processing (NLP) behind it? How have you made it better than other NLP?

Tom Gruber: Siri’s top line measure of success is task completion (not relevance). A subtask is intent recognition, and subtask of that is NLP. Speech is another element, which couples to NLP and adds its own issues. In this context, Siri’s NLP is “pretty darn good” — if the user is talking about something in Siri’s domains of competence, its intent understanding is right the vast majority of the time, even in the face of noise from speech, single finger typing, and bad habits from too much keywordese. All NLP is tuned for some class of natural language, and Siri’s is tuned for things that people might want to say when talking to a virtual assistant on their phone. We evaluate against a corpus, but I don’t know how it would compare to standard message and news corpuses using by the NLP research community.

Nova Spivack: Did you develop your own speech interface, or are you using third-party system for that? How good is it? Is it battle-tested?

Tom Gruber: We use third party speech systems, and are architected so we can swap them out and experiment. The one we are currently using has millions of users and continuously updates its models based on usage.

Nova Spivack: Will Siri be able to talk back to users at any point?

Tom Gruber: It could use speech synthesis for output, for the appropriate contexts. I have a long standing interest in this, as my early graduate work was in communication prosthesis. In the current mobile internet world, however, iPhone-sized screens and 3G networks make it possible to do so more much than read menu items over the phone. For the blind, embedded appliances, and other applications it would make sense to give Siri voice output.

Nova Spivack: Can you give me more examples of how the NLP in Siri works?

Tom Gruber: Sure, here’s an example, published in the Technology Review, that illustrates what’s going on in a typical dialogue with Siri. (Click link to view the table)

Nova Spivack: How personalized does Siri get – will it recommend different things to me depending on where I am when I ask, and/or what I’ve done in the past? Does it learn?

Tom Gruber: Siri does learn in simple ways today, and it will get more sophisticated with time. As you said, Siri is already personalized based on immediate context, conversational history, and personal information such as where you live. Siri doesn’t forget things from request to request, as do stateless systems like search engines. It always considers the user model along with the domain and task models when coming up with results. The evolution in learning comes as users have a history with Siri, which gives it a chance to make some generalizations about preferences. There is a natural progression with virtual assistants from doing exactly what they are asked, to making recommendations based on assumptions about intent and preference. That is the curve we will explore with experience.

Nova Spivack: How does Siri know what is in various external services – are you mining and doing extraction on their data, or is it all just real-time API calls?

Tom Gruber: For its current domains Siri uses dozens of APIs, and connects to them in both realtime access and batch data synchronization modes. Siri knows about the data because we (humans) explicitly model what is in those sources. With declarative representations of data and API capabilities, Siri can reason about the various capabilities of its sources at run time to figure out which combination would best serve the current user request. For sources that do not have nice APIs or expose data using standards like the Semantic Web, we can draw on a value chain of players that do extract structure by data mining and exposing APIs via scraping.

Nova Spivack: Thank you for the information, Siri might actually make me like the iPhone enough to start using one again.

Tom Gruber: Thank you, Nova, it’s a pleasure to discuss this with someone who really gets the technology and larger issues. I hope Siri does get you to use that iPhone again. But remember, Siri is just starting out and will sometimes say silly things. It’s easy to project intelligence onto an assistant, but Siri isn’t going to pass the Turing Test. It’s just a simpler, smarter way to do what you already want to do. It will be interesting to see how this space evolves, how people will come to understand what to expect from the little personal assistant in their pocket.

“All reality is virtual” — Terrence McKenna

This is Part II of my article “The Global Brain is About to Wake Up” — about the  realtime Web and how it relates to the emerging Global Brain.

Here I focus mainly on thorny philosophical and scientific speculations about the nature of consciousness, the role it plays in the universe, and whether or not the Web can ever be said to be conscious in its own right. Beware — this content may not be of interest to most of my readers. It’s certainly in the “wild speculation” category.

Will the Web Become Conscious?

As the realtime Web gets faster and richer, it will begin to appear to be more cohesive and collectively intelligent. It will begin to appear like an actual, unified Global Brain, rather than just a crowd. Instead of being just a collection of interacting parts we will be able to see it as a functioning whole — a kind of entity in its own right. We will also be able to see this collective “entitiness” emerge for subsets of the whole Web? For example will nations, organizations, markets, industries, enterprises, workgroups and teams start to seem more intelligent? The Web will get smarter and faster, at every level of collective cognition but will it ever actually become conscious?

Yes and no.

It will become collectively more intelligent, and the consciousnesses of individuals around the Web will be more connected and potentially synchronized. But the Web itself won’t actually have it’s own new consciousness, unique from the consciousnesses of the people who participate in it. Still it will seem more conscious than it was before, simply by virtue of the human consciousnesses within it being more connected and focused.

I don’t think the Web will actually develop or have its own meta-level consciousness however. It won’t evolve some new form of Web-scale consciousness that is totally separate from the individual consciousness of the people on the Web.  A Web-scale sentient entity that is unique and separate from the humans minds on the Web will never exist. That will never happen. Instead, the Web as a whole will evolve to better utilize the human consciousness that is already present within it — the consciousness that we human beings already have.

The Irreplaceable Role of Humans in the Web

As conscious entities, we humans play a unique and irreplaceable role in the realtime Web and the Global Brain.  We provide the only consciousness the Web will ever have. Machines may be able to sense and measure what is going on, and even make sense of it for us in ways that transcend the abilities of the individual human brain, but they won’t be able to be conscious of what is going on the way that we humans can be.

We human beings are the consciousness of the Web — that is our special role. No machine or set of machines can replicate consciousness, not even the entire Web as a single machine. However there is a distinction to be made between consciousness and intelligence.

Machines can certainly be made to be intelligent, and that applies even to the entire Web as a machine as well. The Web is getting more intelligent, and as this happens it is becoming our Global Brain. But it’s not becoming more conscious.  Rather, we humans are becoming more conscious of the Web and what is going on within it. Humans are becoming able to be more conscious of the Web, but the Web itself is not becoming conscious at all, let alone more conscious. This is a key point to keep in mind.

Until recently humans have been watching the Web in slow motion. We can only see small glimpses at a time. The individual human brain cannot comprehend the vast patterns that are taking place on the Web, and there are few software tools that can make sense of them for us either. It’s just too big and complex a system, and the patterns which comprise its collective thoughts — the thoughts of the Global Brain — are too spread out in time and space.

We humans are barely able to be lucidly conscious in our little nows — which are really just spans of a few square meters, and a few minutes, at a time — but the collectively intelligent processes and patterns out on the Web cover thousands of miles and can span days, weeks, months or even years. They just don’t fit in our little human nows. The solution is to find a way to visualize them so we can digest them in our little nows. That’s the only practical approach — unless someone figures out how to expand the individual human now.

Fortunately, there are several trends that are going help with this process. As the Web gets faster, processes that used to take too long for us to follow them will become short enough for individuals to watch them play out in reasonable timespans, without getting lost or overwhelmed. The collective thoughts of the Web are starting to happen fast enough for our human minds to see them emerge, change, and interact on our human timescale of minutes and hours. Instead of watching memes develop and spread on the Web as if in slow-motion, we are starting to see and measure them in our timescale, at our speed.

In addition as the Web gets more computationally powerful — computers and software will be able to help us see what is going on beyond the limits of our human nows — larger volumes of data changing over larger spans of time than we can grasp on our own. This too will help to compress and visualize patterns and processes that were previously beyond our comprehension in ways that we can make sense of as individual human observers with our small brains and short nows.

Both of these trends will enable individual human minds to comprehend larger and more complex processes and patterns within the Web. And as individuals become able to be conscious of larger and more complex patterns taking place within the Web, they will be able to react and adapt to those patterns in their own individual behavior. This feedback loop will give rise to increasingly intelligent collective adaptation and behavior. And thus the Web as a whole — the Global Brain that includes humans, machines, software, and all our infrastructure — will appear to become increasingly smart.

Humans drive this process by simply being conscious observers of the Web, and by making intelligent decisions, adding content and taking actions online. But we’re not the only ones. Software will also play a role in this — adding intelligence and content, but not consciousness, to the process.

How Important is Consciousness Anyway?

But how important is human consciousness to the Web, and the Global Brain? One might wonder whether human consciousness really matters in all this, or whether it’s enough just to have non-conscious but intelligent machines?

Would the Global Brain be different without humans there to witness it? If there were no humans in it, but just non-conscious artificially intelligent software that simply follows rules or uses statistics and algorithms — would the Global Brain be more or less conscious or intelligent?

This is actually an absurd question. Without humans there would not be a Web, let alone a Global Brain. But let’s just suspend that for a moment and ask the question in a different way. Suppose that at some time in the distant future, all humans die, but the Web remains. Would the Web still contain any consciousness on it’s own?

I think the answer is no. This ultimately goes back to John Searle’s concept of Qualia. In a nutshell, there is nothing on the Web, apart from humans, that is capable of experiencing qualia — the actual knowing of any experience. So there is nothing on the Web that is capable of being conscious, apart from the humans who participate in it. Without the humans, there could be no consciousness in or on the Web.

There  is a difference between being conscious of the qualia of something, and simply measuring data about something. Qualia is special — as strange and potentially hocus-pocus at that may sound. I just don’t believe qualia is something that can be synthesized in a machine or by any algorithm. Being conscious of the Web is not the same as simply measuring data flows. I believe there is a distinct quality of “knowing” or “being aware” that is the hallmark of actual consciousness and which simply cannot be synthesized in a computer.

From what I can tell, qualia is something unique to being sentient, in other words, aware. And awareness is something special as far as I can tell — I think it might be fundamental like space and time, not something we can create or synthesize, and not something emergent. Again this just my opinion — but I think it’s a defensible one.

The Unexplainableness of Consciousness

I’ve spent decades thinking about the question of consciousness, and whether machines can ever be conscious, and I have never found it plausible to make conscious machines.

Quite the contrary — the more I have examined this question, the more clear it has become to me that consciousness is special — it is something that simply cannot even be described, and literally cannot be found — yet it is undeniably taking place. Ontologically consciousness is similar to space and time — we cannot find space or time, we cannot isolate them or grasp their substance, yet they are undeniably taking place. Consciousness seems to be just like that. Unexplainable, yet undeniable.

I’m something of a mystic with regard to consciousness — but not in a blind way. I’ve come to this view only after really trying to avoid it — through very thorough and painstaking investigation from just about every perspective on it — neuroscience, cognitive science, artificial intelligence, linguistics, philosophy, physics, cosmology, and religion and spirituality.

Consciousness appears finally to be something we just cannot explain, let alone synthesize, and I’d be willing to bet that it’s always going to be beyond our reach. In fact I have made such a bet at the Long Bets project: You can read more about this in my article, “Why Machines Will Never be Conscious.”

Given that I view consciousness as something primordial and beyond physics, from my perspective at least, I don’t think we can manufacture it. I also doubt it will simply magically emerge on the Web, apart from individual human minds.

Consciousness is Neither Emergent Nor Reducible.

But wait. Certainly there is a case to be made that if consciousness can emerge within the human brain, then why not within the Web? The human brain is essentially a more complex Web after all. Why is one kind of Web any more or less qualified to be conscious?

My present answer to this is that I don’t think consciousness ever emerges through some physical process — it’s never created or destroyed, and even when said to be present it’s not “there” like other kinds of things. It doesn’t appear as something, it has no form, shape, color, etc. It cannot be found or grasped at all. It is similar to space in these respects.

Space is never created or destroyed — at least as far as we can tell from within this universe. Similarly, consciousness is never created or destroyed as far as we can tell as conscious observers. That’s just how the universe is — it’s a mystery that is bigger than us. We’ll never be able to comprehend it fully from inside it. Consciousness seems to have the same ontological status as space. The difference is that while space is inert, incapable of observing or knowing, consciousness seems to have a quality of knowing that is quite unique.

My point is actually that the human brain is NOT special. I don’t actually think consciousness comes from the brain or is inside the brain, or running like some kind of software on the hardware of the brain.

If consciousness were merely some physical phenomenon that depended on the brain, then it would be no problem to synthesize it, not just for AI but for the Web as a whole as well. But that’s not the case, in my opinion.

I don’t think consciousness is a material thing, nor is it an emergent phenomena. I think it’s fundamental to the nature of the universe — like space and time — or perhaps even more fundamental than space and time. We can’t create it. We’ll probably never fully understand it. It just is there from the start. It’s the very basis of the entire phenomena of the universe, it’s not merely something that evolves and emerges within the universe. Indeed, I would venture to state that without consciousness — at least in primordial form — no universes would even be possible or would ever arise.

In my view, material things like the physical universe and the human body and brain, emerge from consciousness, rather than consciousness emerging from material things. Consciousness, whatever it is, is primordial and fundamental. Whether or not you reify it as a fundamental first-cause or ultimate thing, or you take the Buddhist view that it is also empty of any entity or nature and therefore not a thing, it is still at least totally primordial.

This isn’t just speculation — it’s something that can both be observed and is entirely logical as well. For example, if you really look closely at what you or anyone can possibly ever observe, it appears this is the only tenable answer we can find. Why? Because we cannot observe anything prior to being conscious ourselves — consciousness is necessary to be an observer. We can’t even ask such questions if we are not conscious in the first place. Consciousness is assumed, and must already be there, as soon as we even start looking for it.

Furthermore, the example of dreams proves that incredibly real virtual worlds, entire universes, can indeed appear and take place within the sphere of an individual dreaming consciousness — and they are indistinguishable (while they occur) from waking experience. Dreaming illustrates the power and scope of consciousness — it shows that it is not absurd to think that the our own so-called waking experience could be appearing like a dream within our own fields of consciousness. Waking experience, like dreaming, happens within the sphere of consciousness. It’s impossible to have waking experience without being conscious.

We have no evidence of there being anything beyond the sphere of consciousness and we cannot possibly observe anything without resorting to consciousness in the process to make the observation. There is no way to logically establish that there exists anything beyond or before the scope of consciousness. Anything we attempt to prove or observer is mediated by our own observing consciousness.

For this reason, as far as I or anyone can ever discern, it is reasonable to posit that each of our unique perspectives — each of our minds — contains the universe from one perspective. It’s similar to a hologram — where each piece of the picture contains the whole picture from a different angle. In the case of consciousness, each individual consciousness is one unique perspective on the universe. And the universe itself cannot be found apart from all these conscious perspectives on it. It’s not “out there” as some separate physical thing that these consciousnesses are simply watching from afar — it is literally a manifestation of these consciousnesses, there is no duality between observer and what is observed at the quantum level.

All the evidence points to consciousness being prior to everything else. There is in fact no evidence that indicates otherwise. As a result I don’t believe consciousness is emergent or reducible. I don’t think it is created or destroyed. And even when present it is not actually findable, because it is basically an axiom of the system we are in. It’s primordial and so we cannot sense it or detect it, other than with consciousness itself. There’s nothing more fundamental to break it down into, or to compare or contrast it against.

Consciousness and the Quantum Substrate

From what I can discern so far, I believe that human consciousness — actual sentience, not simulated sentience — is fundamentally related to the fabric of space-time. It is woven right into the quantum substrate of reality.

At that level of reality there is not clear distinction between mind and matter, it’s some kind of whole that we barely understand. While computers may be able to simulate aspects of this, they do not actually interact directly with the quantum substrate the way that human consciousness does.

This is a big difference between machine minds and human minds: Human consciousness is directly connected to the fundamental quantum nature of the universe, and quite probably plays a role in creating or at least conditioning observed reality. Computer programs — no matter how sophisticated — are not connected to the quantum substrate in the same way — they are not capable of being true quantum observers.

There is at least some evidence for my view of consciousness: On a quantum level, observation and measurement seem to have an impact on what is actually found to occur. The observer affects the experiment. All forms of observation eventually seem to require a human — or equivalently conscious — observer at some point in the process — there’s no escaping that. Without such an observer, the universe remains in an indeterminate quantum state. So it appears that human consciousness — or at least authentic actual consciousness whether human or not — is required to cause the quantum field to actually crystallize into particular events.

On the other hand, there is no evidence that computers can ever be conscious; no evidence that synthetic sentient observers can be created, and even if we created them, there would be no way to prove that their powers of observation were equivalent to our own. Any observations they made of them would ultimately be observed by us humans, and so we would always be the final conscious observers in the chain.

On a quantum level, our observation of our machines, would cascade downwards, causing their observations of reality to have an effect. Without our observing them, machines would not be able to actually affect the quantum level of reality. And indeed it would be difficult to try to prove otherwise, because a human observer is necessary to observe any such proof or system we can devise (and in fact, quantum observer effects have been shown even to propagate backwards in time from a later act of observation to an earlier experiment). So there’s just no way to take human consciousness out of the loop.

We cannot prove that human consciousness isn’t necessary for our universe to appear. We cannot prove that machines can function as independent quantum observers, separate from human observation, and we probably cannot devise any experiment or device which could prove that therefore. There’ s really no evidence to suggest that machines could synthesize this function — all the evidence in fact says otherwise. And this applies by extension to the Web as a whole, and thus to the Global Brain.

As a result, I think human consciousnesses play an absolutely crucial role in the universe, the  Web, and in any eventual Global Brain or form of collective intelligence. Our consciousness is the only actual authentic consciousness in the system. And it plays an important and necessary role at a quantum level in shaping reality through quantum level acts of observation.

By the way, it’s worth noting that consciousness is not exclusively the domain of human beings — animals are also conscious for example. But human beings are at least the most intelligent conscious things that we know of, so I’m limiting this discussion of the Global Brain to humans. In any case, there is no substitute for actual consciousness. It can’t be synthesized. It comes only from humans. At best it can perhaps be aimed, funneled or maybe amplified.

But that doesn’t mean that machine intelligence won’t play a very important enabling and catalyzing role in making the Global Brain smarter. There’s a difference between consciousness and intelligence. In fact, machine intelligence is critical to the Global Brain waking up — because it makes the vast complexity of the Global Brain (in both space and time) comprehensible, digestible, and accessible to the individual human consciousnesses that observe it.

Although humans posses consciousness, our minds are limited in scope — we simply cannot see or make sense of patterns that are above a certain level of scale or complexity in space and time. We need help with that — and that’s where computers enter the story, with their vast abilities to calculate, sort, collate, correlate, and organize masses of data.

Computers essentially increase the scope of human consciousness, by enabling us to observe things and do computations that are beyond the abilities of the individual human brain. It is by making the vast patterns within the complex whole — the entire Web –  more visible and understandable to the observers within it — the human consciousnesses within it — that the Global Brain actually becomes smarter, more reflexively-aware, and more collectively conscious.

By connecting individual human consciousnesses to the vast intelligence and knowledge of the growing global computing network, we will get the best of both: a Global Brain that gets increasingly collectively aware and intelligent.

The emerging realtime Web is not only going to speed up the Web and our lives, it is going to bring about a kind of awakening of our collective Global Brain. It’s going to change how many things happen on online, but it’s also going to change how we see and understand what the Web is doing. By speeding up the Web, it will cause processes that used to take weeks or months to unfold online, to happen in days or even minutes. And this will bring these processes to the human-scale — to the scale of our human “now” — making it possible for us to be aware of larger collective processes than before. We have until now been watching the Web in slow motion. As it speeds up, we will begin to see and understand what’s taking place on the Web in a whole new way.

This process of of quickening is part of a larger trend which I and others call “Nowism.” You can read more of my thoughts about Nowism here. Nowism is an orientation that is gaining momentum and will help to shape this decade, and in particular, how the Web unfolds. It is the idea that the present-timeframe (“the now”) is getting more important, shorter and also more information-rich. As this happens our civilization is becoming more focused on the now, and less focused on past or the future. Simply keeping up with the present is becoming an all-consuming challenge: Both a threat and an opportunity.

The realtime Web –  what I call “The Stream”  (see “Welcome to the Stream”) — is changing the unit of now. It’s making it shorter. The now is the span of time which we have to be aware of to be effective our work and lives, and it is getting shorter. On a personal level the now is getting shorter and denser — more information and change is packed into shorter spans of time; a single minute on Twitter is overflowing with potentially relevant messages and links. In business as well, the now is getting shorter and denser — it used to be about the size of a fiscal quarter, then it became a month, then a week, then a day, and now it is probably about half a day in span. Soon it will be just a few hours.

To keep up with what is going on we have to check in with the world in at least half-day chunks. Important news breaks about once or twice a day. Trends on Twitter take about a day to develop too. So basically, you can afford to just check  the news and the real-time Web once or twice a day and still get by. But that’s going to change.  As the now gets shorter, we’ll have to check in more frequently to keep abreast of change. As the Stream picks up speed in the middle of this decade, to remain competitive will require near-constant monitoring — we will have to always be connected to, and watching, the real-time Web and our personal streams. Being offline at all will risk missing out on big important trends, threats and opportunities that emerge and develop within minutes or hours. But nobody is capable of tracking the Stream all 24/7 — we must at least take breaks to eat and sleep. And this is a problem.

Big Changes to the Web Coming Soon…

With Nowism comes a faster Web, and this will lead to big changes in how we do various activities on the Web:

• We will spend less time searching. Nowism pushes us to find better alternatives to search, or to eliminate search entirely, because people don’t have time to search anymore. We need tools that do the searching for us and that help with decision support so we don’t have to spend so much of our scarce time doing that. See my article on “Eliminating the Need for Search — Help Engines” for more about that.
• Monitoring (not searching) the real-time stream becomes more important. We need to stay constantly vigilant about what’s happening, what’s trending. We need to be alerted of the important stuff (to us), and we need a way to filter out what’s not important to us. Probably a filter based on influence of people and tweets, and/or time dynamics of memes will be necessary. Monitoring the real-time stream effectively is different from searching it. I see more value in real-time monitoring than realtime search — I haven’t seen any monitoring tools for Twitter that are smart enough to give me just the content I want yet. There’s a real business opportunity there.
• The return of agents. Intelligent agents are going to come back. To monitor the realtime Web effectively each of us will need online intelligent agents that can help us — because we don’t have time, and even if we did, there’s just too much information to sift through.
• Influence becomes more important than relevance. Advertisers and marketers will look for the most influential parties (individuals or groups) on Twitter and other social media to connect with and work through. But to do this there has to be an effective way to measure influence. One service that’s providing a solution for this (which I’ve angel invested in and advise) is Klout.com – they measure influence per person per topic. I think that’s a good start.
• Filtering content by influence. We also will need a way to find the most influential content. Influential content could be the content most RT’d or most RT’d by most influential people. It would be much less noisy to be able to see only the more influential tweets of people I follow. If a tweet gets RT’d a lot, or is RT’d by really influential people, then I want to see it. If not, then only if it’s really important (based on some rule). This will be the only way to cope with the information overload of the real-time Web and keep up with it effectively. I don’t know of anyone providing a service for this yet. It’s a business opportunity.
• Nowness as a measure of value of content. We will need a new form of ranking of results by “nowness” – how timely they are now. So for example, in real-time search engines we shouldn’t rank results merely by how recent they are, but also by how timely, influential, and “hot” they are now. See my article from years ago on “A Physics of Ideas” for more about that. Real-time search companies should think of themselves as real-time monitoring companies — that’s what they are really going to be used for in the end. Only the real-time search ventures that think of themselves this way are going to survive the conceptual paradigm shift that the realtime Web is bringing about. In a realtime context, search is actually too late — once something has happened in the past it really is not that important anymore –what matters is current awareness: discovering the trends NOW. To do that one has to analyze the present, and the very recent past, much more than searching the longer term past. The focus has to be on real-time or near-real-time analytics, statistical analysis, topic and trend detection, prediction, filtering and alerting. Not search.
• New ways to understand and navigate the now. We will need a way to visualize and navigate the now. I’m helping to incubate a stealth startup venture, Live Matrix, that is working on that. It hasn’t launched yet. It’s cool stuff. More on that in the future when they launch.
• New tools for browsing the Stream. New tools will emerge for making the realtime Web more compelling and smarter. I’m working on incubating some new stealth startups in this area as well. They’re very early-stage so can’t say more about them yet.
• The merger of semantics with the realtime Web. We need to make the realtime Web semantic — as well as the rest of the Web — in order to make it easier for software to make sense of it for us. This is the best approach to increasing the signal-to-noise ratio of content we have to look at whether searching or monitoring stuff. The Semantic Web standars of the W3C are key to this. I’ve written a long manifesto on this in “Minding The Planet: The Meaning and Future of the Semantic Web” if you’re really interested in that topic.

Faster Leads to Smarter

As the realtime web unfolds and speeds up, I think it will also have a big impact on what some people call “The Global Brain.” The Global Brain has always existed, but in recent times it has been experiencing a series of major upgrades — particularly around how connected, affordable, accessible and fast it is. First we got phone and faxes, then the Internet, the PC and the Web, and now the real-time Web and the Semantic Web. All of these recent changes are making the Global Brain faster, more richly interconnected. And this makes it smarter. For more about my thoughts on the Global Brain, see these two talks:

• My detailed History and Future of the Global Brain given at the Singularity Summit.
• A talk on the emerging Global Brain and human-machine cybernetic superorganism, with specific focus on what it means for media companies, from the GRID Conference.

What’s most interesting to me is that as the rate of communication and messaging on the Web approaches near-real time, we may see a kind of phase change take place – a much smarter Global Brain will sort of begin to appear out of the chaos. In other words, the speed of collective thinking is as important to the complexity or sophistication of collective thinking, in making the Global Brain significantly more intelligent. In other words, I’m proposing that there is a sort of critical speed of collective thinking, before which the Global Brain seems like just a crowd of actors chaotically flocking around memes, and after which the Global Brain makes big leaps — instead of seeming like a chaotic crowd, it starts to look more like an organized group around certain activitities — it is able to respond to change faster, and optimize and even do things collectively more productively than a random crowd could.

This is kind of like film, or animation. When you watch a movie or animation you are really watching a rapid series of frames. This gives the illusion of there being cohesive, continuous characters, things and worlds in the movie — but really they aren’t there at all, it’s just an illusion — our brains put these scenes together and start to recognize and follow higher order patterns. A certain shape appears to maintain itself and move around relative to other shapes, and we name it with a certain label — but there isn’t really something there, let alone something moving or interacting — there are just frames flicking by rapidly . It turns out that after a critical frame rate (around 20 to 60 frames per second) the human brain stops seeing individual frames and starts seeing a continuous movie. When you start flipping pages fast enough it appears to be a coherent animation and then we start seeing things “moving within the sequence” of frames. In the same way, as the unit of time of (aka the speed) of the real-time Web increases, its behavior will start to seem more continuous and smarter — we won’t see separate chunks of time or messages, we’ll see intelligent continuous collective thinking and adaptation processes.

In other words, as the Web gets faster, we’ll start to see processes emerge within it that appear to be cohesive intelligent collective entities in their own right. There won’t really be any actual entities there that we can isolate, but when we watch the patterns on the Web it will appear as if such entities are there. This is basically what is happening at every level of scale — even in the real world. There really isn’t anything there that we can find — everything is divisible down to the quantum level and probably beyond — but over time our brains seem to recognize and label patterns as discrete “things.” This is what will happen across the Web as well. For example, a certain meme (such as a fad or a movement) may become a “thing” in it’s own right, a kind of entity that seemingly takes on a life of its own and seems to be doing something. Similarly certain groups or social networks or activities they engage in may seem to be intelligent entities in their own rights.

This is an illusion in that there really are no entities there, they are just collections of parts that themselves can be broken down into more parts, and no final entities can be found. However, nonethless, they will seem like intelligent entities when not analyzed in detail. In addition, the behavior of these chaotic systems may resist reduction — they may not even be understandable and their behavior may not be predictable through a purely reductionist approach — it may be that they react to their own internal state and their environments virtually in real-time, making it difficult to take a top-down or bottom-up view of what they are doing. In a realtime world, change happens in every direction.

As the Web gets faster, the patterns that are taking place across it will start to become more animated. Big processes that used to take months or years to happen will happen in minutes or hours. As this comes about we will begin to see larger patterns than before, and they will start to make more sense to us — they will emerge out of the mists of time so to speak, and become visible to us on our human timescale — the timescale of our human-level “now. As a result, we will become more aware of higher order dynamics taking place on the real-time Web, and we will begin to participate in and adapt to those dynamics, making those dynamics in turn even smarter. (For more on my thoughts about how the Global Brain gets smarter, see:  “How to Build the Global Mind.”)

See Part II: “Will The Web Become Conscious?” if you want to dig further into the thorny philosophical and scientific issues that this brings up…

We are so focused on how to improve present-day search engines. But that is a kind of mental myopia. In fact, a more interesting and fruitful question is why do people search at all? What are they trying to accomplish? And is there a better way to help them accomplish that than search?

Instead of finding more ways to get people to search, or ways to make existing search experiences better, I am starting to think about how to reduce or  eliminate the need to search — by replacing it with something better.

People don’t search because they like to. They search because there is something else they are trying to accomplish. So search is in fact really just an inconvenience — a means-to-an-end that we have to struggle through to do in order to get to what we actually really want to accomplish. Search is “in the way” between intention and action. It’s an intermediary stepping stone. And perhaps there’s a better way to get to where we want to go than searching.

Searching is a boring and menial activity. Think about it. We have to cleverly invent and try pseudo-natural-language queries that don’t really express what we mean. We try many different queries until we get results that approximate what we’re looking for. We click on a bunch of results and check them out. Then we search some more. And then some more clicking. Then more searching. And we never know whether we’ve been comprehensive, or have even entered the best query, or looked at all the things we should have looked at to be thorough. It’s extremely hit or miss. And takes up a lot of time and energy. There must be a better way! And there is.

Instead of making search more bloated and more of a focus, the goal should really be get search out of the way.  To minimize the need to search, and to make any search that is necessary as productive as possible. The goal should be to get consumers to what they really want with the least amount of searching and the least amount of effort, with the greatest amount of confidence that the results are accurate and comprehensive. To satisfy these constraints one must NOT simply build a slightly better search engine!

Instead, I think there’s something else we need to be building entirely. I don’t know what to call it yet. It’s not a search engine. So what is it?

Bing’s term “decision engine” is pretty good, pretty close to it. But what they’ve actually released so far still looks and feels a lot like a search engine. But at least it’s pushing the envelope beyond what Google has done with search. And this is good for competition and for consumers. Bing is heading in the right direction by leveraging natural language, semantics, and structured data. But there’s still a long way to go to really move the needle significantly beyond Google to be able to win dominant market share.

For the last decade the search wars have been fought in battles around index size, keyword search relevancy, and ad targeting — But I think the new battle is going to be fought around semantic understanding, intelligent answers, personal assistance, and commerce affiliate fees. What’s coming next after search engines are things that function more like assistants and brokers.

Wolfram Alpha is an example of one approach to this trend. The folks at Wolfram Alpha call their system a “computational knowledge engine” because they use a knowledge base to compute and synthesize answers to various questions. It does a lot of the heavy lifting for you, going through various data, computing and comparing, and then synthesizes a concise answer.

There are also other approaches to getting or generating answers for people — for example, by doing what Aardvark does: referring people to experts who can answer their questions or help them. Expert referral, or expertise search, helps reduce the need for networking and makes networking more efficient. It also reduces the need for searching online — instead of searching for an answer, just ask an expert.

There’s also the semantic search approach — perhaps exemplified by my own Twine “T2″ project — which basically aims to improve the precision of search by helping you get to the right results faster, with less irrelevant noise. Other consumer facing semantic search projects of interest are Goby and Powerset (now part of Bing).

Still another approach is that of Siri, which is making an intelligent “task completion assistant” that helps you search for and accomplish things like “book a romantic dinner and a movie tonight.” In some ways Siri is a “do engine” not a “search engine.” Siri uses artificial intelligence to help you do things more productively. This is quite needed and will potentially be quite useful, especially on mobile devices.

All of these approaches and projects are promising. But I think the next frontier — the thing that is beyond search and removes the need for search is still a bit different — it is going to combine elements of all of the above approaches, with something new.

For a lack of a better term, I call this a “help engine.” A help engine proactively helps you with various kinds of needs, decisions, tasks, or goals you want to accomplish. And it does this by helping with an increasingly common and vexing problem: choice overload.

The biggest problem is that we have too many choices, and the number of choices keeps increasing exponentially. The Web and globalization have increased the number of choices that are within range for all of us, but the result has been overload. To make a good, well-researched, confident choice now requires a lot of investigation, comparisons, and thinking. It’s just becoming too much work.

For example, choosing a location for an event, or planning a trip itinerary, or choosing what medicine to take, deciding what product to buy, who to hire, what company to work for, what stock to invest in, what website to read about some topic. These kinds of activities require a lot of research, evaluations of choices, comparisons, testing, and thinking. A lot of clicking. And they also happen to be some of the most monetizable activities for search engines. Existing search engines like Google that make money from getting you to click on their pages as much as possible have no financial incentive to solve this problem — if they actually worked so well that consumers clicked less they would make less money.

I think the solution to what’s after search — the “next Google” so to speak — will come from outside the traditional search engine companies. Or at least it will be an upstart project within one of them that surprises everyone and doesn’t come from the main search teams within them. It’s really such a new direction from traditional search and will require some real thinking outside of the box.

I’ve been thinking about this a lot over the last month or two. It’s fascinating. What if there was a better way to help consumers with the activities they are trying to accomplish than search? If it existed it could actually replace search. It’s a Google-sized opportunity, and one which I don’t think Google is going to solve.

Search engines cause choice overload. That wasn’t the goal, but it is what has happened over time due to the growth of the Web and the explosion of choices that are visible, available, and accessible to us via the Web.

What we need now is not a search engine — it’s something that solves the problem created by search engines. For this reason, the next Google probably won’t be Google or a search engine at all.

I’m not advocating for artificial intelligence or anything that tries to replicate human reasoning, human understanding, or human knowledge. I’m actually thinking about something simpler. I think that it’s possible to use computers to provide consumers with extremely good, automated decision-support over the Web and the kinds of activities they engage in. Search engines are almost the most primitive form of decision support imaginable. I think we can do a lot better. And we have to.

People use search engines as a form of decision-support, because they don’t have a better alternative. And there are many places where decision support and help are needed: Shopping, travel, health, careers, personal finance, home improvement, and even across entertainment and lifestyle categories.

What if there was a way to provide this kind of personal decision-support — this kind of help — with an entirely different user experience than search engines provide today? I think there is. And I’ve got some specific thoughts about this, but it’s too early to explain them; they’re still forming.

I keep finding myself thinking about this topic, and arriving at big insights in the process. All of the different things I’ve worked on in the past seem to connect to this idea in interesting ways. Perhaps it’s going to be one of the main themes I’ll be working on and thinking about for this coming decade.

This is the story of Twine.com — our early research (with never before seen screenshots of our early semantic desktop work), and our evolution from Twine 1.0 towards Twine 2.0 (“T2″) which is focused on semantic search.

This is a talk I have given many times, on the past, present and future evolution of the Web, and particularly the Semantic Web.

I have noticed an interesting and important trend of late. The Web is starting to spread outside of what we think of as “the Web” and into “the World.” This trend is exemplified by many data points. For example:

• The Web on mobile devices like the iPhone. Finally it’s really usable on a phone. Now it goes everywhere with us. Soon we will track our own paths on our phones as we move around, creating a virtual map of our favorite places and routes.
• Location aware applications and services, such as Google Maps Mobile. They link physical places to virtual places on the Web.
• The Web in cars.  Auto avigation units will soon be Web-enabled.
• Next-generation Wi-Fi digital cameras are wifi-enabled, linking directly to camera GPS and to photo sharing and storage services. Will cloud-centric wireless cameras with zero local storage come next?
• Web picture frames such as Ceiva bring the Web into your grandma’s livingroom.
• The Web in restaurants and stores. Your server gets your reservation on the Web from OpenTable. In-store kiosks connect to the Web to help you shop, or to bring up your online account and shopping cart.
• The Web in your garden. GardenGro‘s sensor connects your garden to the Web, in order to figure out what to plant and how to cultivate it in your actual location.
• Everything becomes trackable with RFID. Physical objects have virtual locations.
• Sensors are connecting to the Web and popping up everywhere. For example here.
• Plastic Logic‘s portable plastic reading device. The pad of paper, version 2.0.
• The beginnings of an Internet of Things — where every thing has an address on the Web.
• The rise of Lifestreaming, in which everything (or much of what) one does is captured to the Web and even broadcast.
• Progress on Augmented Reality — instead of the physical world going into virtual worlds, the virtual world is going to flow into the physical world.

These are just a few data points. There are many many more. The trendline is clear to me.

Things are not going to turn out the way we thought. Instead of everything going digital — a future in which we all live as avatars in cyberspace — The digital world is going to invade the physical world. We already are the avatars and the physical world is becoming cyberspace. The idea that cyberspace is some other place is going to dissolve because everything will be part of the Web. The digital world is going physical.

When this happens — and it will happen soon, perhaps within 20 years or less — the notion of “the Web” will become just a quaint, antique concept from the early days when the Web still lived in a box. Nobody will think about “going on the Web” or “going online” because they will never NOT be on the Web, they will always be online.

Think about that. A world in which every physical object, everything we do, and eventually perhaps our every thought and action is recorded, augmented, and possibly shared. What will the world be like when it’s all connected? When all our bodies and brains are connected together — when even our physical spaces, furniture, products, tools, and even our natural environments, are all online? Beyond just a Global Brain, we are really building a Global Body.

The World is becoming the Web. The “Web Wide World” is coming and is going to be a big theme of the next 20 years.

In typical Web-industry style we’re all focused minutely on the leading trend-of-the-year, the real-time Web. But in this obsession we have become a bit myopic. The real-time Web, or what some of us call “The Stream,” is not an end in itself, it’s a means to an end. So what will it enable, where is it headed, and what’s it going to look like when we look back at this trend in 10 or 20 years?

In the next 10 years, The Stream is going to go through two big phases, focused on two problems, as it evolves:

1. Web Attention Deficit Disorder. The first problem with the real-time Web that is becoming increasingly evident is that it has a bad case of ADD. There is so much information streaming in from so many places at once that it’s simply impossible to focus on anything for very long, and a lot of important things are missed in the chaos. The first generation of tools for the Stream are going to need to address this problem.
2. Web Intention Deficit Disorder. The second problem with the real-time Web will emerge after we have made some real headway in solving Web attention deficit disorder. This second problem is about how to get large numbers of people to focus their intention not just their attention. It’s not just difficult to get people to notice something, it’s even more difficult to get them to do something. Attending to something is simply noticing it. Intending to do something is actually taking action, expending some energy or effort to do something. Intending is a lot more expensive, cognitively speaking, than merely attending. The power of collective intention is literally what changes the world, but we don’t have the tools to direct it yet.

The Stream is not the only big trend taking place right now. In fact, it’s just a strand that is being braided together with several other trends, as part of a larger pattern. Here are some of the other strands I’m tracking:

• Messaging. The real-time Web aka The Stream is really about messaging in essence. It’s a subset of the global trend towards building a better messaging layer for the Web. Multiple forms of messaging are emerging, from the publish-and-subscribe nature of Twitter and RSS, to things like Google Wave, Pubsubhubub, and broadcast style messaging or multicasting via screencast, conferencing and media streaming and events in virtual worlds. The effect of these tools is that the speed and interactivity of the Web are increasing — the Web is getting faster. Information spreads more virally, more rapidly — in other words, “memes” (which we can think of as collective thoughts) are getting more sophisticated and gaining more mobility.
• Semantics. The Web becomes more like a database. The resolution of search, ad targeting, and publishing increases. In other words, it’s a higher-resolution Web. Search will be able to target not just keywords but specific meaning. For example, you will be able to search precisely for products or content that meet certain constraints. Multiple approaches from natural language search to the metadata of the Semantic Web will contribute to increased semantic understanding and representation of the Web.
• Attenuation. As information moves faster, and our networks get broader, information overload gets worse in multiple dimensions. This creates a need for tools to help people filter the firehose. Filtering in its essence is a process of attenuation — a way to focus attention more efficiently on signal versus noise. Broadly speaking there are many forms of filtering from automated filtering, to social filtering, to personalization, but they all come down to helping someone focus their finite attention more efficiently on the things they care about most.
• The WebOS.  As cloud computing resources, mashups, open linked data, and open API’s proliferate, a new level of aggregator is emerging. These aggregators may focus on one of these areas or may cut across them. Ultimately they are the beginning of true cross-service WebOS’s. I predict this is going to be a big trend in the future — for example instead of writing Web apps directly to various data and API’s in dozens of places, just write to a single WebOS aggregator that acts as middleware between your app and all these choices. It’s much less complicated for developers. The winning WebOS is probably not going to come from Google, Microsoft or Amazon — rather it will probably come from someone neutral, with the best interests of developers as the primary goal.
• Decentralization. As the semantics of the Web get richer, and the WebOS really emerges it will finally be possible for applications to leverage federated, Web-scale computing. This is when intelligent agents will actually emerge and be practical. By this time the Web will be far too vast and complex and rapidly changing for any centralized system to index and search it. Only massively federated swarms of intelligent agents, or extremely dynamic distributed computing tools, that can spread around the Web as they work, will be able to keep up with the Web.
• Socialization. Our interactions and activities on the Web are increasingly socially networked, whether individual, group or involving large networks or crowds. Content is both shared and discovered socially through our circles of friends and contacts. In addition, new technologies like Google Social Search enable search results to be filtered by social distance or social relevancy. In other words, things that people you follow like get higher visibility in your search results. Socialization is a trend towards making previously non-social activities more social, and towards making already-social activities more efficient and broader. Ultimately this process leads to wider collaboration and higher levels of collective intelligence.
• Augmentation. Increasingly we will see a trend towards augmenting things with other things. For example, augmenting a Web page or data set with links or notes from another Web page or data set. Or augmenting reality by superimposing video and data onto a live video image on a mobile phone. Or augmenting our bodies with direct connections to computers and the Web.

If these are all strands in a larger pattern, then what is the megatrend they are all contributing to? I think ultimately it’s collective intelligence — not just of humans, but also our computing systems, working in concert.

Collective Intelligence

I think that these trends are all combining, and going real-time. Effectively what we’re seeing is the evolution of a global collective mind, a theme I keep coming back to again and again. This collective mind is not just comprised of humans, but also of software and computers and information, all interlinked into one unimaginably complex system: A system that senses the universe and itself, that thinks, feels, and does things, on a planetary scale. And as humanity spreads out around the solar system and eventually the galaxy, this system will spread as well, and at times splinter and reproduce.

But that’s in the very distant future still. In the nearer term — the next 100 years or so — we’re going to go through some enormous changes. As the world becomes increasingly networked and social the way collective thinking and decision making take place is going to be radically restructured.

Social Evolution

Existing and established social, political and economic structures are going to either evolve or be overturned and replaced. Everything from the way news and entertainment are created and consumed, to how companies, cities and governments are managed will change radically. Top-down beaurocratic control systems are simply not going to be able to keep up or function effectively in this new world of distributed, omnidirectional collective intelligence.

Physical Evolution

As humanity and our Web of information and computatoins begins to function as a single organism, we will evolve literally, into a new species: Whatever is after the homo sapien. The environment we will live in will be a constantly changing sea of collective thought in which nothing and nobody will be isolated. We will be more interdependent than ever before. Interdependence leads to symbiosis, and eventually to the loss of generality and increasing specialization. As each of us is able to draw on the collective mind, the global brain, there may be less pressure on us to do things on our own that used to be solitary. What changes to our bodies, minds and organizations may result from these selective evolutionary pressures? I think we’ll see several, over multi-thousand year timescales, or perhaps faster if we start to genetically engineer ourselves:

• Individual brains will get less good at things like memorization and recall, calculation, reasoning, and long-term planning and action.
• Individual brains will get better at multi-tasking, information filtering, trend detection, and social communication. The parts of the nervous system involved in processing live information will increase disproportionately to other parts.
• Our bodies may actually improve in certain areas. We will become more, not less, mobile, as computation and the Web become increasingly embedded into our surroundings, and into augmented views of our environments. This may cause our bodies to get into better health and shape since we will be less sedentary, less at our desks, less in front of TV’s. We’ll be moving around in the world, connected to everything and everyone no matter where we are. Physical strength will probably decrease overall as we will need to do less manual labor of any kind.

These are just some of the changes that are likely to occur as a result of the things we’re working on today. The Web and the emerging Real-Time Web are just a prelude of things to come.

The next generation of Web search is coming sooner than expected. And with it we will see several shifts in the way people search, and the way major search engines provide search functionality to consumers.

Web 1.0, the first decade of the Web (1989 – 1999), was characterized by a distinctly desktop-like search paradigm. The overriding idea was that the Web is a collection of documents, not unlike the folder tree on the desktop, that must be searched and ranked hierarchically. Relevancy was considered to be how closely a document matched a given query string.

Web 2.0, the second decade of the Web (1999 – 2009), ushered in the beginnings of a shift towards social search. In particular blogging tools, social bookmarking tools, social networks, social media sites, and microblogging services began to organize the Web around people and their relationships. This added the beginnings of a primitive “web of trust” to the search repertoire, enabling search engines to begin to take the social value of content (as evidences by discussions, ratings, sharing, linking, referrals, etc.) as an additional measurment in the relevancy equation. Those items which were both most relevant on a keyword level, and most relevant in the social graph (closer and/or more popular in the graph), were considered to be more relevant. Thus results could be ranked according to their social value — how many people in the community liked them and current activity level — as
well as by semantic relevancy measures.

In the coming third decade of the Web, Web 3.0 (2009 – 2019), there will be another shift in the search paradigm. This is a shift to from the past to the present, and from the social to the personal.

Established search engines like Google rank results primarily by keyword (semantic) relevancy. Social search engines rank results primarily by activity and social value (Digg, Twine 1.0, etc.). But the new search engines of the Web 3.0 era will also take into account two additional factors when determining relevancy: timeliness, and personalization.

Google returns the same results for everyone. But why should that be the case? In fact, when two different people search for the same information, they may want to get very different kinds of results. Someone who is a novice in a field may want beginner-level information to rank higher in the results than someone who is an expert. There may be a desire to emphasize things that are novel over things that have been seen before, or that have happened in the past — the more timely something is the more relevant it may be as well.

These two themes — present and personal — will define the next great search experience.

To accomplish this, we need to make progress on a number of fronts.

First of all, search engines need better ways to understand what content is, without having to do extensive computation. The best solution for this is to utilize metadata and the methods of the emerging semantic web.

Metadata reduces the need for computation in order to determine what content is about — it makes that explicit and machine-understandable. To the extent that machine-understandable metadata is added or generated for the Web, it will become more precisely searchable and productive for searchers.

This applies especially to the area of the real-time Web, where for example short “tweets” of content contain very little context to support good natural-language processing. There a little metadata can go a long way. In addition, of course metadata makes a dramatic difference in search of the larger non-real-time Web as well.

In addition to metadata, search engines need to modify their algorithms to be more personalized. Instead of a “one-size fits all” ranking for each query, the ranking may differ for different people depending on their varying interests and search histories.

Finally, to provide better search of the present, search has to become more realtime. To this end, rankings need to be developed that surface not only what just happened now, but what happened recently and is also trending upwards and/or of note. Realtime search has to be more than merely listing search results chronologically. There must be effective ways to filter the noise and surface what’s most important effectively. Social graph analysis is a key tool for doing this, but in
addition, powerful statistical analysis and new visualizations may also be required to make a compelling experience.

DRAFT 1 — A Work in Progress

Introduction

Here’s an idea I’ve been thinking about: it’s a concept for a new philosophy, or perhaps just a name for a grassroots philosophy that seems to be emerging on its own. It’s called “Nowism.” The view that now is what’s most important, because now is where one’s life actually happens.

Certainly we have all heard terms like Ram Das’ famous, “Be here now” and we may be familiar with the writings of Eckhart Tolle and his “Power of Now” and others. In addition there was the “Me generation” and the more recent idea of “living in the now.” On the Web there is also now a growing shift towards real-time, what I call the Stream.

These are all examples of the emergence of this trend. But I think these are just the beginnings of this movement — a movement towards a subtle but major shift in the orientation of our civilization’s collective attention. This is a shift towards the now, in every dimension of our lives. Our personal lives, professional lives, in business, in government, in technology, and even in religion and spirituality.

I have a hypothesis that this philosophy — this worldview that the “now” is more important than the past or the future, may come to characterize this new century we are embarking on. If this is true, then it will have profound effects on the direction we go in as a civilization.

It does appear that the world is becoming increasingly now-oriented; more real-time, high-resolution, high-bandwidth. The present moment, the now, is getting increasingly flooded with fast-moving and information-rich streams of content and communication.

As this happens we are increasingly focusing our energy on keeping up with, managing, and making sense of, the now. The now is also effectively getting shorter — in that more happens in less time, making the basic clockrate of the now effectively faster. I’ve written about this elsewhere.

Given that the shift to a civilization that is obsessively focused on the now is occurring, it is not unreasonable to wonder whether this will gradually penetrate into the underlying metaphors and worldviews of coming generations, and how it might manifest as differences from our present-day mindsets.

How might people who live more in the now differ from those who paid more attention to the past, or the future? For example, I would assert that the world in and before the 19th century was focused more on the past than the now or the future. The 20th century was characterized by a shift to focus more on the future than the past or the now. The 21st century will be characterized by a shift in focus onto the now, and away from the past and the future.

How might people who live more in the now think about themselves and the world in coming decades. What are the implications for consumers, marketers, strategists, policymakers, educators?

With this in mind, I’ve attempted to write up what I believe might be the start of a summary of what this emerging worldview of “Nowism” might be like.

It has implications on several levels: social, economic, political, and spiritual.

Nowism Defined

Like Buddhism, Taoism, and other “isms,” Nowism is a view on the nature of reality, with implications for how to live one’s life and how to interpret and relate to the world and other people.

Simply put: Nowism is the philosophy that the span of experience called “now” is fundamental. In other words there is nothing other than now. Life happens in the now. The now is what matters most.

Nowism does not claim to be mutually exclusive with any other religion. It merely claims that all other religions are contained within it’s scope — they, like everything else, take place exclusively within the now, not outside it. In that respect the now, in its actual nature, is fundamentally greater than any other conceivable philosophical or religious system, including even Nowism itself.

Risks of Unawakened Nowism

Nowism is in some ways potentially short-sighted in that there is less emphasis on planning for the future and correspondingly more emphasis on living the present as fully as possible. Instead of making decisions with their effects in the future foremost in mind, the focus is on making the optimal immediate decisions in the context of the present. However, what is optimal in the present may not be optimalover longer spans of time and space.

What may be optimal in the now of a particular individual may not at all be optimal in the nows of other individuals. Nowism can therefore lead to extremely selfish behavior that actually harms others, or it can lead to extremely generous behavior on a scale that far transcends the individual, if one strives to widen their own experience of the now sufficiently.

Very few individuals will ever do the necessary work to develop themselves to the point where their actual experience of now is dramatically wider than average. It is however possible to do this, while quite rare. Such individuals are capable of living exclusively in the now while still always acting with the long-term benefit of both themselves all other beings in mind.

The vast majority of people however will tend towards a more limited and destructive form of Nowism, in which they get lost in deeper forms of consumerism, content and media immersion, hedonism, and conceptualization. Rather than being freed by the now, they will be increasingly imprisoned by it.

This lower form of Nowism — what might be called unawakened Nowism — is characterized by an intense focus on immediate self-gratification, without concern or a sense of responsibility for the consequences of one’s actions on oneself or others in the future. This kind of living in the moment, while potentially extremely fun, tends to end badly for most people. Fortunately most people outgrow this tendency towards extremely unawakened Nowism after graduating college and/or entering the workforce.

Abandoning extremely unawakened Nowist lifestyles doesn’t necessarily result in one realizing any form of awakened Nowism. One might simply remain in a kind of dormant state, sleepwalking through life, not really living fully in the present, not fully experiencing the present in all its potential. To reach this level of higher Nowism, or advanced Nowism, one must either have a direct spontaneous experience of awakening to the deeper qualities of the now, or one must study, practice and work with teachers and friends who can help them to reach such a direct experience of the now.

Benefits of Awakened Nowism: Spiritual and Metaphysical Implications of Nowist Philosophy

In the 21st Century, I believe Nowism may actually become an emerging movement. With it there will come a new conception of the self, and of the divine. The self will be realized to be simultaneously more empty and much vaster than was previously thought. The divine will be understood more directly and with less conceptualization. More people will have spiritual realization this way, because in this more direct approach there is less conceptual material to get caught up in. The experience of now is simply left as it is — as direct and unmediated, unfettered, and unadulterated as possible.

This is a new kind of spirituality perhaps. One in which there is less personification of the divine, and less use of the concept of a personified deity as an excuse or justification for various worldy actions (like wars and laws, for example).

Concepts about the nature of divinity have been used by humans for millenia as tools for various good and bad purposes. But in Nowism, these concepts are completely abandoned. This also means abandoning the notion that there is or is not a divine nature at the core of reality, and each one of us. Nowists do not get caught up in such unresolvable debates. However, at the same time, Nowists do strive for a direct realization of the now — one that is as unmediated and nonconceptual as possible — and that direct realization is considered to BE thedivine nature itself.

Nowism does not assert that nothing exists or that nothing matters. Such views are nihilism not Nowism. Nowism does not assert that what happens is caused or uncaused — such views are those of the materialists and the idealists, not Nowism. Instead Nowism asserts the principles of dependent origination, in which cause and-effect appears to take place, even though it is an illusory process and does not truly exist. On the basis of a relative-level cause-effect process, an ethical system can be founded which seeks to optimize happiness and minimize unhappiness for the greatest number of beings, by adjusting ones actions so as to create causes that lead to increasingly happy effects for oneself and others, increasingly often. Thus the view of Nowism does not lead to hedonism — in fact, anyone who makes a careful study of the now will reach the conclusion that cause and effect operates unfailingly and therefore is a key tool for optimizing happiness in the now.

Advanced Nowists don’t ignore cause-and-effect, in fact quite the contrary: they pay increasingly close attention to cuase-and-effect and their particular actions. The natural result is that they begin to live a life that is both happier and that leads to more happiness for all other beings — at least this is the goal and example of the best-case. The fact that cause-and-effect is in operation, even though it is notfundamentally real, is the root of Nowist ethics. It is precisely the same as the Buddhist conception of the identity of emptiness and dependent-origination.

Numerous principles follow from the core beliefs of Nowism. They include practical guidance for living ones life with a minimum of unnecessary suffering (of oneself as well as others), further principles concerning the nature of reality and the mind, and advanced techniques and principles for reaching greater realizations of the now.

As to the nature of what is taking place right now: from the Nowist perspective, it is beyond concepts, for all concepts, like everything else, appear and disappear like visions or mirages, without ever truly-existing. This corresponds precisely to the Buddhist conception of emptiness.

The scope of the now is unlimited, however for the uninitiated the now is usually considered to be limited to the personal present experience of the individual. Nowist adepts, on the other hand, assert that the scope of the now may be modified (narrowed or widened) through various exercises including meditation, prayer, intense physical activity, art, dance and ritual, drugs, chanting, fasting, etc.

Narrowing the scope of the now is akin to reducing the resolution of present experience. Widening the scope is akin to increasing the resolution. A narrower now is a smaller experience, with less information content. A wider now is a larger experience, with more information content.

Within the context of realizing that now is all there is, one explores carefully and discovers that now does not contain anything findable (such as a self, other, or any entity or fundamental basis for any objective or subjective phenomenon, let alone any nature that could be called “nowness” or the now itself).

In short the now is totally devoid of anything findable whatsoever, although sensory phenomena do continue to appear to arise within it unceasingly. Such phenomena, and the sensory apparatus, body, brain, mind and any conception of self that arises in reaction to them, are all merely illusion-like appearances with no objectively-findable ultimate, fundamental, or independent existence.

This state is not unlike the analogy of a dream in which oneself and all the other places and characters are all equally illusory, or of a completely immersive virtual reality experience that is so convincing one forgets it isn’t real.

Nowism does not assert a divine being or deity, although it also is not mutually exclusive with the existence of one or more such beings. However all such beings are considered to be no more real than any other illusory appearance, such as the appearances of sentient beings, planets, stars, fundamental particles, etc. Any phenomena — whether natural or supernatural — are equally empty of any independent true existince. They are all illusory in nature.

However, Nowists do assert that the nature of the now itself, while completely empty, is in fact the nature of consciousness and what we call life. It cannot be computed, simulated or modeled in an information system, program, machine, or representation of any kind. Any such attempts to represent the now are merely phenomena appearing within the now, not the now itself. The now is fundamentally transcendental in this respect.

The now is not limited to any particular region in space or time, let alone to any individual being’s mind. There is no way to assert there is a single now, or many nows, for no nows are actually findable.

The now is the gap between the past and the future, however, when searched for it cannot really be found, nor can the past or future be found. The past is gone, the future hasn’t happened yet, and the now is infinite, constantly changing, and ungraspable. The entire space-time continuum is in fact within a total all-embracing now, the cosmically extended now that is beyond the limited personalized scope of now we presently think we have. Through practice this can be gradually glimpsed and experienced to greater degrees.

As the now is explored to greater depths, one begins to find that it has astonishing implications. Simultaneously much of the Zen literature — especially the koans — starts to make sense at last.

While Nowism could be said to be a branch of Buddhism, I would actually say it might be the other way arond. Nowism is really the most fundamental, pure, philosophy — stripped of all cultural baggage and historical concepts, and retaining only what is absolutely essential.

May 8, 2009

Welcome to The Stream

The Internet began evolving many decades before the Web emerged. And while today many people think of the Internet and the Web as one and the same, in fact they are different. The Web lives on top of the Internet’s infrastructure much like software and documents live on top of an operating system on a computer.

And just as the Web once emerged on top of the Internet, now something new is emerging on top of the Web: I call this the Stream. The Stream is the next phase of the Internet’s evolution. It’s what comes after, or on top of, the Web we’ve all been building and using.

Perhaps the best and most current example of the Stream is the rise of Twitter, Facebook and other microblogging tools. These services are visibly streamlike, their user-interfaces are literally streams; streams of ideas, thinking and conversation. In reaction to microblogs we are also starting to see the birth of new tools to manage and interact with these streams, and to help understand, search, and follow the trends that are rippling across them. Just as the Web is not any one particular site or service, the Stream is not any one site or service — it’s the collective movement that is taking place across them all.

To meet the challenges and opportunities of the Stream a new ecosystem of services is rapidly emerging: stream publishers, stream syndication tools, stream aggregators, stream readers, stream filters, real-time stream search engines, and stream analytics engines, stream advertising networks, and stream portals are emerging rapidly. All of these new services are the beginning of the era of the Stream.

Web History

The original Tim Berners-Lee proposal that started the Web was in March, 1989. The first two decades of the Web (Web 1.0 from 1989 – 1999, and Web 2.0 from 1999 – 2009) were focused on the development of the Web itself. Web 3.0 (2009 – 2019), the third-decade of the Web, officially began in March of this year and will be focused around the Stream.

• In the 1990′s with the advent of HTTP and HTML, the metaphor of “the Web” was born and concepts of webs and sites captured our imaginations.
• In the early 2000′s the focus shifted to graphs such as social networks and the beginnings of the Semantic Web.
• Now, in the coming third decade, the focus is shifting to the Stream and with it, stream oriented metaphors of flows, currents, and ripples.

The Web has always been a stream. In fact it has been a stream of streams. Each site can be viewed as a stream of pages developing over time. Each page can be viewed as a stream of words, that changes whenever it is edited. Branches of sites can also be viewed as streams of pages developing in various directions.

But with the advent of blogs, feeds, and microblogs, the streamlike nature of the Web is becoming more readily visible, because these newer services are more 1-dimensional and conversational than earlier forms of websites, and they update far more frequently.

Defining the Stream

Just as the Web is formed of sites, pages and links, the Stream is formed of streams.

Streams are rapidly changing sequences of information around a topic. They may be microblogs, hashtags, feeds, multimedia services, or even data streams via APIs.

The key is that streams change often. This change is an important part of the value they provide (unlike static Websites, which do not necessarily need to change in order to provide value). In addition, it is important to note that streams have URI’s — they are addressable entities.

So what defines a stream versus an ordinary website?

1. Change. Change is the key reason why a stream is valuable. That is not always so with a website.  Websites do not have to change at all to be valuable — they could for example just be static but comprehensive reference library collections. But streams on the other hand change very frequently, and it is this constant change that is their main point.
2. Interface Independence.
   Streams are streams of data, and they can be fully accessed and consumed independently of any particular user-interface — via syndication of their data into various tools. Websites on the other hand, are only accessible via their user-interfaces. In the era of the Web the provider controlled the interface. In the new era of the stream, the consumer controls the interface.
3. Conversation is king.
   An interesting and important point is that streams are linked together not by hotlinks, but by acts of conversation — for example, replies, “retweets,” comments and ratings, and “follows.” In the era of the Web the hotlink was king. But in the era of the Stream conversation is king.

In terms of structure, streams are comprised of agents, messages and interactions:

• Agents are people as well as software apps that publish to streams.
• Messages are publications by agents to streams — for example, short posts to their microblogs.
• Interactions are communication acts, such as sending a direct message or a reply, or quoting someone (“retweeting”), that connect and transmit messages between agents.

The Global Mind

If the Internet is our collective nervous system, and the Web is our collective brain, then the Stream is our collective mind. The nervous system and the brain are like the underlying hardware and software, but the mind is what the system is actually thinking in real-time. These three layers are interconnected, yet are distinctly different aspects, of our emerging and increasingly awakened planetary intelligence.

The Stream is what the Web is thinking and doing, right now. It’s our collective stream of consciousness.

The Stream is the dynamic activity of the Web, unfolding over time. It is the conversations, the live streams of audio and video, the changes to Web sites that are happening, the ideas and trends — the memes — that are rippling across millions of Web pages, applications, and human minds.

The Now is Getting Shorter

The Web is changing faster than ever, and as this happens, it’s becoming more fluid. Sites no longer change in weeks or days, but hours, minutes or even seconds. if we are offline even for a few minutes we may risk falling behind, or even missing something absolutely critical. The transition from a slow Web to a fast-moving Stream is happening quickly. And as this happens we are shifting our attention from the past to the present, and our “now” is getting shorter.

The era of the Web was mostly about the past — pages that were published months, weeks, days or at least hours before we looked for them. Search engines indexed the past for us to make it accessible: On the Web we are all used to searching Google and then looking at pages from the recent past and even farther back in the past. But in the era of the Stream, everything is shifting to the present — we can see new
posts as they appear and conversations emerge around them, live, while we watch.

Yet as the pace of the Stream quickens, what we think of as “now” gets shorter. Instead of now being a day, it is an hour, or a few minutes. The unit of change is getting more granular.

For example, if you monitor the public timeline, or even just your friends timeline in Twitter or Facebook you see that things quickly flow out of view, into the past. Our attention is mainly focused on right now: the last few minutes or hours. Anything that was posted before this period of time is “out of sight, out of mind.”

The Stream is a world of even shorter attention spans, online viral sensations, instant fame, sudden trends, and intense volatility. It is also a world of extremly short-term conversations and thinking.

This is the world we may be entering. It is both the great challenge, and the great opportunity of the coming decade of the Web.

How Will We Cope With the Stream?

The Web has always been a stream — it has been happening in real-time since it started, but it was slower — pages changed less frequently, new things were published less often, trends developed less quickly. Today it is getting so much faster, and as this happens its feeding back on itself and we’re feeding into it, amplifying it even more.

Things have also changed qualitatively in recent months. The streamlike aspects of the Web have really moved into the foreground of our mainstream cultural conversation. Everyone is suddenly talking about Facebook and Twitter. Celebrities. Talk show hosts. Parents. Teens.

And suddenly we’re all finding ourselves glued to various activity streams, microblogging manically and squinting to catch fleeting references to things we care about as they rapidly flow by and out of view. The Stream has arrived.

But how can we all keep up with this ever growing onslaught of information effectively? Will we each be knocked over by our own personal firehose, or will tools emerge to help us filter our streams down to managable levels? And if we’re already finding that we have too many streams today, and must jump between them ever more often, how will we ever be able to function with 10X more streams in a few years?

Human attention is a tremendous bottleneck in the world of the Stream. We can only attend to one thing, or at most a few things, at once. As information comes at us from various sources, we have to jump from one item to the next. We cannot absorb it all at once. This fundamental barrier may be overcome with technology in the future, but for the next decade at least it will still be a key obstacle.

We can follow many streams, but only one-item-at-a-time; and this requires rapidly shifting our focus from one article to another and from one stream to another. And there’s no great alternative: Cramming all our separate streams into one merged activity stream quickly gets too noisy and overwhelming to use.

The ability to view different streams for different contexts is very important and enables us to filter and focus our attention effectively. As a result, it’s unlikely there will be a single activity stream — we’ll have many, many streams. And we’ll have to find ways to cope with this reality.

Streams may be unidirectional or bidirectional. Some streams are more like “feeds” that go from content providers to content consumers. Other streams are more like conversations or channels in which anyone can be both a provider and a consumer of content.

As streams become a primary mode of content distribution and communication, they will increasingly be more conversational and less like feeds. And this is important — because to participate in a feed you can be passive, you don’t have to be present synchronously.  But to participate in a conversation you have to be present and synchronous — you have to be there, while it happens, or you may miss out on it entirely.

A Stream of Challenges and Opportunities

We are going to need new kinds of tools for managing and participating in streams, and we are already seeing the emergence of some of them. For example Twitter clients like Tweetdeck, RSS feed readers, and activity stream tracking tools like Facebook and Friendfeed. There are also new tools for filtering our streams around interests, for example Twine.com (* Disclosure: the author of this article is a principal in Twine.com). Real-time search tools are also emerging to provide quick ways to scan the Stream as a whole. And trend discovery tools are helping us to see
what’s hot in real-time.

One of the most difficult challenges will be how to know what to pay attention to in the Stream: Information and conversation flow by so quickly that we can barely keep up with the present, let alone the past. How will know what to focus on, what we just have to read, and what to ignore or perhaps read later?

Recently many sites have emerged that attempt to show what is trending up in real-time, for example by measuring how many retweets various URLs are getting in Twitter. But these services only show the huge and most popular trends. What about all the important stuff that’s not trending up massively? Will people even notice things that are not widely RT’d or “liked”? Does popularity equal importance of content?

Certainly one measure of the value of an item in the Stream is social popularity. Another measure is how relevant it is to a topic, or even more importantly, to our own personal and unique interests. To really cope with the Stream we will need ways to filter that combine both these different approaches. Furthermore as our context shifts throughout the day (for example from work to various projects or clients to shopping to health to entertainment, to family etc) we need tools that can adapt to filter the Stream differently based on what we now care about.

A Stream oriented Internet also offers new opportunities for monetization. For example, new ad distribution networks could form to enable advertisers to buy impressions in near-real time across URLs that are trending up in the Stream, or within various slices of it. For example, an advertiser could distribute their ad across dozens of pages that are getting heavily retweeted right now. As those pages begin to decline in RT’s per minute, the ads might begin to move over to different URLs that are starting to gain.

Ad networks that do a good job of measuring real-time attention trends may be able to capitalize on these trends faster and provide better results to advertisers. For example, an advertiser that is able to detect and immediately jump on the hot new meme of the day, could get their ad in front of the leading influencers they want to reach, almost instantly. And this could translate to sudden gains in awareness and branding.

The emergence of the Stream is an interesting paradigm shift that may turn out to characterize the next evolution of the Web, this coming third-decade of the Web’s development. Even though the underlying data model may be increasingly like a graph, or even a semantic graph, the user experience will be increasingly stream oriented.

Whether Twitter, or some other app, the Web is becoming increasingly streamlike. How will we filter this stream? How will we cope? Whoever can solve these problems first and best is probably going to get rich.

Other Articles on This Topic

http://www.techmeme.com/090517/p6#a090517p6

http://www.techcrunch.com/2009/05/17/jump-into-the-stream/

http://www.techcrunch.com/2009/02/15/mining-the-thought-stream/

(DRAFT 2. A Work-In-Progress)

The Problem: Our Communities are Failing

I’ve been thinking about community lately. There is a great need for a new and better model for communities in the world today.

Our present communites are not working and most are breaking down or stagnating. Cities are experiencing urbanization and a host of ensuing  social and economic challenges. Meanwhile the movement towards cities has drained the people — particularly young professionals — away from rural communities, causing them to stagnate and decline.

Local economies have been challenged by national and global economic integratio — from outsourcing of jobs away to other places, to giant retail chains such as Walmart swooping in and driving out local businesses.

From giant megacities and multi-city urban sprawls, to inner city neighborhoods, to suburban bedroom communities, and rural towns and villages, the pain is being felt everywhere and at all levels.

Our current models for community don’t scale, they don’t work anymore, and they don’t fit the kind of world we are living in today. And why should they? After all, they were designed a long time ago for a very different world.

At the same time there are increasing numbers of singles or couples without children, and even families and neighborhoods that are breaking down as cities get larger.

The need for community is growing not declining — especially as existing communities fail and no other alternatives take their place. Loneliness, social isolation, and social fragmentation are huge and growing problems — they lead to crime, suicide, mental illness, lack of productivity, moral decay, civil unrest, and just about every other social and economic problem there is.

The need for an updated and redesigned model for community is increasingly important to all of us.

Intentional Communities

In particular, I am thinking about intentional communities — communities in which people live geographically near one another, and participate in community together, by choice. They may live together or not, dine together or not, work together or not, worship together or not — but at least they need to live within some limit of proximity to one another and participate in community together. These are the minimum requirements.

But is there a model that works? Or is it time to design a new model that fits the time and place in which we live better?

Is this simply a design problem that we can solve by adopting the right model, or is there something about human nature that makes it impossible to succeed no matter what model we apply?

I am an optimist and I don’t think human nature prevents healthy communities from forming and being sustainable. I think it’s a design problem. I think this problem can (and must) be solved with a set of design principles that work better than the ones we’ve come up with so far. This would be a great problem to solve. It could even potentially improve the lives of billions of people.

Models of Intentional Community

Community is extremely valuable and important. We are social beings. And communities enable levels of support and collaboration, economic growth, resiliance, and perhaps personal growth, that individuals or families cannot achieve on their own.

However, do intentional communities work? What examples can we look at and what can we glean from them about what worked and what didn’t?

All of the cities and towns in the world started as intentional communities but today many seem to have lost their way as they got larger or were absorbed into larger communities.

As for smaller intentional communities — recent decades are littered with all kinds of spectacular failures.

The communes and experiemental communities of the 1960′s and 1970′s have mostly fallen apart.

Spiritual communities seem to either tend towards becoming personality cults that are highly prone to tyrranny and corruption, or they too seem to fall apart eventually as well.

There have been so many communities around various gurus, philosophers, or cult-figures, but they have almost all universally become cults or have broken apart.

Human nature is hard to wrangle without strong leadership, yet strong leadership and the power it entails leads inevitably to ego and corruption.

At least some ashrams in India seem to be working well, although their internal dynamics are usually centered around a single guru or leadership group — and while there may be a strong social agreement within these communities, this is not a model of community that will work for everyone. And in fact, only in extremely rare cases, are there any gurus who are actually selfless enough to hold that position without abusing it.

Other kinds of religious communities are equally prone to problems — however perhaps at least some, such as the Quakers, Shakers, and Amish may have solved this — I am not sure however. If they were so successful, why are there so few of them?

Temporary communities are another type of intentional community, for example, Burning Man, seem to work quite well, but only for temporary periods of time — they would have the same problems of all other communities if they became institutionalized or tried to not be temporary.

Educational communities, such as university towns and campuses, do appear to work in many cases. They combine both an ongoing community (tenured faculty, staff and townspeople) and temporary communities (seasonal student and faculty residents).

Economic communes — such as the communes in Soviet-era Russia were prone to corruption, and failed as economic experiments. In Soviet Russia “some were more equal than others” and that ultimately led to corruption and tyranny.

Political-economic communities such as the neighborhood groups in Maoist China only worked because they were firmly, even brutally, controlled from the central government. They were not exactly voluntary intentional communities.

I don’t know enough about the Israeli Kibbutzim experiments, but they at least seem to be continuing, although I am not sure how well they function — I admit my ignorance on that topic.

One type of intentional community that does seem to work are caregiving communities such as assisting living communities, nursing homes, halfway houses, etc — but perhaps they seem to work only because their members don’t remain very long.

Why Aren’t There More Intentional Communities?

So here is my question: Do intentional communities work? And if they work so well, why aren’t there more of them? Or are they flourishing and multiplying under the radar?

Is there a model (or are there models) for intentional community that have proven long-term success? Where are the examples?

Is the fact that there are not more intentional communities emerging and thriving, evidence that intentional communities just don’t work or have stopped replicating or evolving? Or is it evidence that the communities we already live in work well enough, even though they are no longer intentional for most of us?

I don’t think our present-day communities work well enough, nor are they very healthy or rewarding to their participants. I do believe there is the possibility, and even the opportunity, to come up with a better model — one which works so well that it attracts people, grows and self-replicates around the world rapidly. But I don’t yet know what that new model is.

Design Principles

To design the next-evolution of intentional community, perhaps we can start with a set of design principles gleaned from what we have learned from existing communities?

This set of design principles should be selected to be practical for the world we live in today — a world of rapid transit, economic and social mobility, urban sprawls, cultural and ethnic diversity, cheap air travel, declining birth rates, the 24-7 work week, the Internet, and the globally interdependent economy.

In thinking about this further there are a few key “design principles” which seem to be necessary to make a successful, sustainable, healthy community.

This is not an exhaustive list, but it is what we have thought of so far:

Shared intention.
There has to be a common reason for the group of people to be together. The participants each have to share a common intention to form and participate in a community around common themes and purposes together.

Shared contribution . The participants have to each contribute in various ways to the community as part of their membership.

Shared governance.
The participants each have a role to play in the process of decision making, policy formation, dispute resolution, and operations of the community.

Shared boundaries. There are shared, mutually agreed upon and mutually enforced rules.

Freedom to leave. Anyone can leave the community at any time without pressure to remain.

Freedom of choice.
While in the community people are free to make choices about their roles and participation in the community, within the communities boundaries and governance process. This freedom of choice also includes the freedom to opt out of any role or rule, but that might have the consequence of voluntarily recusing oneself from further participation in the community.

Freedom of expression. The ability for community members to freely and fearlessly express their opinions within the community is an essential element of healthy communities. Systems need to be designed to support and channel this activity. If it is restrained it seeks out other channels anyway (subversion, revolution, etc.). By not restraining expression, but instead desiging a community process that authentically engages members in conversation with one another, the
community can be more self-aware and creativity and innovation can flow more freely.

Representative democratic leadership. The leadership is either by consensus and includes everyone equally, or there is a democratic representative process of electing leaders and making decisions.

Community mobility. This is an interesting topic. In the world today, each person may have different sets of interests and purposes, and they are not all compatible. It may be necessary or desirable to be a member of different communities in different places, times of the year, or periods of one’s life. It
should be possible to be able to be in more than one community, or to rotate through communities, or to change communities as one’s interests, goals, needs and priorities shift over time — so long as one participates in each community fully while they are there. The concept of timesharing in various communities, or what one friend calls “colonies,” is interesting. One might be a member of different colonies — one for their religious interests, one for social kinship, one for a hobby, one for recreation and vacation, etc. These might be in different places and have different members and their role and level of participation might be different in each one. Rather than living in only one particular community, perhaps we need a model where there is more mobility.

Size limitations. One thing I would suggest is that communities work better when they are smaller. The reason for this is that once communities reach a size where each member no longer can maintain a personal relationship with each other member, they stop working and begin to fragment into subgroups. So perhaps limiting the size of a community is a good idea. Or alternatively, when a community reaches a certain size it spawns a new separate community where further growth can happen and all new members go there. In fact, you could even see two communities spawning a new “child” community together to absorb their growth.

Proximity. Communities don’t require that people live near each other — they can function non-locally, for example online. However, the kind of intentional communities I am interested in here are ones where people do live together or near one another, at least part of the time. For this kind of community people need to live and/or dine and/or work together on a periodic, if not frequent basis. An eating co-op in a metropolitan area is an example — at least if everyone has to live within a certain distance and eat together a few times a week, and work a few hours in
the co-op per month. A food co-op, such as co-op grocery store is another example.

Shared Economic Participation. For communities to function there needs to be a form of common currency (either created by the community or from a larger economy the community is situated within), and there should be a form of equitable sharing of collective costs and profits among the community members. There are different ways to distribute the wealth — everyone can be equal no matter what, or reward can be proportional to role, or reward can be proportional to level of contribution, etc. What economic works best in the long-term, for both creating sustainability and growth, for maintaining social order and social justice, and for preventing corruption?

Agility. Communities must be designed to change in order to adapt to new environmental, economic and social realities. Communities that are too rigid in structure or process, or even location, are like species of animals that are unable to continue evolving — and that usually leads to extinction. Part of being agile is being open to new ideas and opportunities. Agility is not just the ability to recognize and react to emerging threats, it is the ability to recognize and react to emerging opportunities as well.

Resiliance. Communities must be designed to be resiliant — Challenges and even damages and setbacks are inevitable. They can be minimized and mitigated, but they will still happen to various degrees. Therefore the design should not assume they can be prevented entirely, but rather should plan for the ability to heal and eventually restore the community as effectively as possible when they do.

Diversity. There are many types of diversity: diversity of opinion, ethnic diversity, age group diversity, religious diversity. Not all communities need to support all kinds of diversity, however it is probably safe to say that for a community to be healthy it must at least support diversity of beliefs and opinions among the membership. No matter what selection criteria is used, there must still be freedom
of thought and belief, and expression, within that group. Communities must be designed to support this diversity, and even encourage it. They also must be designed to manage and process the conversations, conflicts, and changes that diversity brings about. Diversity is a key ingredient that powers growth, agility, and resiliance. In biology diversity is essential to species-survival — mutations are key to evolution. Communities must be designed to mutate, and to intelligently filter in or out those mutations that help or harm the community. Processes that encourange and process diversity are essential for this to happen.