The next Web 2.0 Conference will be upon us in early November and things are busier than ever in the Web 2.0 world.  Along the way, I've managed to miss the one year anniversary of this blog, which I began back in late September of last year.  There have been over 2.5 million direct hits on this site since inception, a large percentage of it due to my Web 2.0 lists such as last year's Best Web 2.0 Software List , but I also get e-mail frequently from die-hard readers as well.  Most importantly however, from all my conversations with people all over the world, it's clear that Web 2.0 remains more than ever a topic of major popular interest and industry fascination.

While the general understanding of Web 2.0 is improving all the time, we have a ways to go before we have a concise, generally accepted definition.  My favorite is still networked applications that explicitly leverage network effects. But while most of what we ascribe to the Web 2.0 name falls out of these definition, it's fairly hard for most of us to extrapolate meaningful ramifications from this.

People that read this blog know that I'm in the camp of folks that try to look beyond Ajax and the visual site design aspect of Web 2.0, and try to capture the deeper design patterns and business models that seem to be powering the most successful Web sites and online companies today.  Though concepts such as harnessing collective intelligence and Data as the Next Intel Inside, as described by Tim O'Reilly , most directly capture the spirit of the Web 2.0 era, it does seem to me that there are a few other elements that we haven't nailed down yet.

At the AjaxWorld Conference and Expo earlier this month, I gave my usual talk about how to formally leverage Web 2.0, with plenty of examples coming from things happening out on the Web.  If you accept that it's the power and size of the Web today , particularly the number of highly interactive network nodes (who are mostly people), give them extremely low-barrier tools, and we should be able to find plenty examples of emergent behavior; significant events happening suddenly and unexpectedly.  Tipping points are getting easier and easier to reach as site designers learn how to create better network effect triggers, draw large audiences suddenly, and as those same audiences increasingly self-organize spontaneously, such as in the KatrinaList project (suddenly) or Wikipedia (slower but bigger).

And it's the arrival of Web 2.0 "supersites" like YouTube , which appear suddenly, often riding the coattails of other major Web 2.0 site's ecosystems, and apply aggressive, viral network effects that show us the true, full scale of the possibilities.  Building a Web site worth over one billion dollars in 18 months is a very impressive result, but it's really only a single axis upon which Web 2.0 can be applied successfully.  Another axis upon which to apply Web 2.0 focuses less on pulling in every single user possible with a horizontal network effect, but on building a difficult to reproduce but highly valuable data source, such as the Navteq mapping database, or Zillow's real estate database.  One might argue that these are still very horizontal but these are merely just well known examples.

The variety and depth of the Web is such that not every Web 2.0 site will have tens of millions of users, nor should it.  An effective Web 2.0 site is largely powered by its users, whose feedback and contributions, direct and indirect, make the site a living ecosystem that evolves from day to day, a mosaic as rich and varied as a sites users would like it to be.  In other words, creating a high quality architectures of participation is becoming a strategic competitive advantage in many areas.

I'm often asked, particularly after one of my presentations on Web 2.0, to articulate the most important and effective actions a site designer can take to realize the benefits of Web 2.0.  As a result, I've created the list below in a attempt to catpure a good, general purpose overview of what these steps are.  My plan in the near future, is to dive into each one of these as much as time permits and explain how they make highly effective Web 2.0 sites not only effective, but often possible at all.  In the meantime, please take them for what they're worth, I believe however that they are instrumental in making a Web site or application the most successful possible.

The Essentials of Leveraging Web 2.0 

• Ease of Use is the most important feature of any Web site, Web application, or program.
  
• Open up your data as much possible. There is no future in hoarding data, only controlling it.
• Aggressively add feedback loops to everything.  Pull out the loops that don’t seem to matter and emphasize the ones that give results.
• Continuous release cycles.  The bigger the release, the more unwieldy it becomes (more dependencies, more planning, more disruption.)  Organic growth is the most powerful, adaptive, and resilient.
• Make your users part of your software.  They are your most valuable source of content, feedback, and passion.  Start understanding social architecture.  Give up non-essential control.  Or your users will likely go elsewhere.
• Turn your applications into platforms. An application usually has a single predetermined use while a platform is designed to be the foundation of something much bigger.  Instead of getting a single type of use from your software and data, you might get hundreds or even thousands of additional uses.
• Don’t create social communities just to have them. They aren’t a checklist item.  But do empower inspired users to create them.
  

Of course, there a lot of work in the details and these are just some of the important, general essentials.  Unfortunately, a lot of careful thinking, planning, and engineering goes into any effective Web 2.0 site and it's having these ideas at the core of it, which can help you get the best results.

Final Note:  I'll be on the road the next two weeks and will be at the Web 2.0 Conference in San Francisco from Nov. 7th-9th.  I'll be there writing coverage for the Web 2.0 Journal and here as much as possible.  If you're going to be there, please drop me a line if you'd like to meet. 

It's the final day of the three day long Web 2.0 Summit , the leading confab for the Web 2.0 era.  It's been a bustling and busy three days in San Francisco with sessions and discussions on a wide variety of Web 2.0 topics, from Advertising 2.0 and Net Neutrality, to the World of Warcraft and Enterprise 2.0.  Given that the Web 2.0 Summit is an executive level conference, the discussion of business models and company strategies around Web 2.0 has dominated the conversation and not the specific techniques and approaches for actually designing and implementing Web 2.0 services and products.  Those subjects have been moved to the upcoming Web 2.0 Expo next April, which will be a much larger event expo-style conference at Moscone Center.

The leadup to the conference was John Musser's great 100-page update of the famous five page Web 2.0 description from Tim O'Reilly (John's comments on the new report here) and the conference also had an exciting Launchpad event to unveil a series of interesting new Web 2.0 sites.  Richard MacManus has the details with links to the sites here on ZDNet.

The two topics that seemed to come up the most often these last three days was 1) how existing major players on the Web can continue on in their leadership roles without significant changes in their business strategies and 2) the need for Web sites and platforms to be as open as possible in order to draw the broadast range of audience and adoption.  In a profiled afternoon conversation on day two, AOL's Jonathan Miller seemed to clearly understand these issues -- which are actively facing his company today -- as it heads into the world of user generated contact and social networks, two forces that are growing large new Web startups, and hence competition, very rapidly.  These new fast growth site models , such as the ones used with YouTube and MySpace, are not however providing clear paths for way for public company to please their investors (net revenue.) Miller also observed that many large companies are not in a position to acquire hot properties like Google did with YouTube.

In another public conversation in the main ballroom right after Jonathan Miller was Microsoft's Ray Ozzie delved into the issues that Microsoft is facing, that started out by focusing on the challenge of how to adapt Microsoft's flagship operating system product, Vista -- as well as their most profitable product, Office -- more effectively to the Web.  The Internet, particularly with Web 2.0 sites, has become the pre-eminent new "superplatform" and it's a significant challenge to Microsoft to stay relevant in a world where the browser is increasingly the center of attention for the software experience.  Ray seemed sanguine about the opportunities however and Vista certainly has many features, such as pervasive built-in syndication, that will certainly pull Vista closer to the Web.

But it was openness that was clearly the most prevalent topic, with discussions on how companies should free their content and services to be used a wider range of situations, particularly from 3rd party entities, even forming the foundation of other products and services offered by entirely different companies.  Openness can also take many forms, from syndicating content to providing well-defined and monetized Web service APIs, and if you don't provide a technical and legal basis for doing so, challenges will only increase as the limited numbers of ways that content and services will reduce the number of overall business opportunities available.  And it puts companies that don't do this at a competitive disadvantage to companies that do open up.  Finally, openness creates the potential for unintended uses, particular as small, more focused content is opened up (smaller chunks are more reusable and general purpose).  It was clear in many discussions, such as with Jonathan Miller, that it's well understood that walled gardens just aren't a viable online business model any longer.

As culled from Web 2.0 Summit discussions and other known best practices...

1. Liberate content and services via a public, open API.  Content will continue to be separated from the experiences that mediate access to it, this makes adaptable experiences possible. Example: RSS readers let users consume content in the ways they choose and have control over.  Doing this turns your Web application into a platform and is one of the most important habits of highly effective Web sites .
2. Syndicate as well as use Web services to open up data. Each method has clear strengths such as discoverability, ease of consumption, or on-demand control.  Example: This means RSS or Atom as well as REST or SOAP.
3. Make it legal to reuse content.  Don't charge if you can help it, consider monetizing it via advertising, transaction fees, or subscriptions.  Don't cripple unintended uses, such as Yahoo!'s limits on their APIs, vs. Amazon's profitable emphasis on unlimited use.
4. Diligently build trust and credibility.  No one will use your open data or services unless there is trust and credibility in the site.  This is very hard to establish and is easily lost.  This is one of the hardest intangibles of openness to manage.
5. Expect the unexpected.  Opening up a site means that others will dream of ways of using your data and services in ways you couldn't imagine.  Often this means they'll use it as a free resource to achieve something that wasn't possible before in terms of scale or volume.  Be prepared for extreme situations and be sure to monitor your feeds and open services and be prepared to throttle them for mailicous or inadvertant waste.

There were plenty of other good sessions at the show and I attended one of the best ones late on the morning of the first day, a great talk from IBM about enterprise mashups, situational software , SOA, and Web 2.0, which are all colliding and combining to make it easier for companies to clear out their application backlog.

Also see great coverage by Stowe Boyd, Richard MacManus , and I hate to say it but ValleyWag, who has relentlessly live blogged the conference.

Announcing Web 2.0 University 

Finally, and pardon the shameless self-promotion, we did have our own big news at the Web 2.0 Summit, namely that O'Reilly Media -- the company that coined the term "Web 2.0" and described the trend to the world -- and my firm, Hinchcliffe & Company, jointly announced on Wednesday that we've formed strategic partnership to join forces on a series of premier services around Web 2.0.  You can read a complete overview of our first major new service, which is already available, called Web 2.0 University. We believe this full series of education and consulting solutions around Web 2.0 will bring intensive, hands-on services around the specific design patterns and business models of Web 2.0.  The premise is that companies are increasingly becoming aware that they need to apply Web 2.0 models to the core of their existing products and services and these services will help them get there quickly and with a minimum of disruption.  So far, early interest has been intriguingly high.

The Web 2.0 Expo will be upon us before we know it.  The deadline for proposals is tomorrow, November 10th, so get them in if you're interested in presenting. See you there!

I was traveling most of last week and so was unable to weigh in on the Web 3.0 mini-tempest that occurred when John Markoff published his exploratory piece in the NY Times last Sunday.  The premise of the article is that we are finding new ways to mine human intelligence which can be exploited by building a new layer of "meaning" on top of the accumulating mass of global collective intelligence that is growing by leaps and bounds every day on the Internet.  Collective intelligence of course is one key aspects of Web 2.0, namely an Internet that is continually improved by constant and sustained contact with hundreds of millions of users contributing content.  These users can either contribute explicitly via a conscious act or implicitly by their very interaction with the Web which then leaves behind useful behavioral "tracks" that can be fed back into the system.  In this ways, hundreds of millions of people are adding to what we know every day, even if individuals contributions are often minor.

Markoff's description of Web 3.0 was ostensibly prompted by something I'm seeing as well, well beyond pure play Web mashups we're beginning to witness a number of companies building end-user solutions that can automatically navigate the Internet, weave together tapestries of online information to generate new, useful results. They can even take it a step beyond: dynamically generated situational Web applications that fully interact with the Web ecosystem.  Such applications -- self-assembled by these tools -- can perform useful tasks such as planning vacations, managing personal schedules, or even orchestrating complex, collaborative business processes for example including entire real-world projects.  The vision is stunning and futuristic yet and the rich fabric of the Web today, with hundreds of open APIs and even vaster reservoirs of content and raw data, now opens the door to the possibility.

Background Reading: Take a look at eight end-user mashup platforms available today 

I've written a lot recently about the trend of user generated software, applications developed by end-users that use the openness of the Web 2.0 era to interact with high value Web services.  But already we're beginning to see the emergence of the next step beyond that: applications developed and tasks completed intelligently by software itself.  Tim-Berners Lee himself envisioned this as the coming Semantic Web which he brilliantly espoused in Scientific American a few years back and has been the goal of great many companies ever since, but which has been relatively unsuccessful on a large scale even up until now.  The reasons for this are complex but seem to lie in what we learned from Web 1.0; a priori solutions often aren't the right ones, emergent ones are .

So while many might say that the 1,200+ mashups currently listed in the trend graphs on Programmable Web are mostly NOT user generated, one only has to look at the widespread use of badges and widgets on MySpace and other major social networking sites to see that everyday people are getting more and more comfortable with "turfing" their blogs and spaces with content, code, and feeds from elsewhere on the Web.  So while much of the end-user mashup activity we see today is probably shallow and don't represent sophisticated functionality, the new tools we're seeing every day are getting better and better and allowing users to take it deeper, creating a true mashup ecosystem.

The shortage of developers and application backlogs: Not finding the app you need

Here's an significant fact, if you look at the number of professional software developers out there today, they are dwarfed by the number of end-users with the time and motivation to describe the solutions that they need.  And interestingly, the same population is dwarfed by the potential output of computer systems that can be directed to create the applications or carry out the tasks we need, with minimal continuous attention on our part.

If you only look at the enterprise IT space you will see that users usually have a long list of things for which they'd like software solutions, but can't get satisfied by the traditional purchase or build processes in place in most organizations.  Every CIO out there is painfully aware of this application backlog but hasn't had the tools to address it.  And out on the Web, there's a different problem: Lots of Web sites, but little software that will do the specific things that a users needs to get accomplished.  As Steve Borch says , "sit back, relax, and let your customers create your products."

Like IBM is realizing with their exploration of end-user driven development products like QEDWiki, most of us today are already conducting much, if not most, of our software integration manually, by re-entering or cutting and pasting data endlessly between our applications.  This implies that 1) there's demand but not enough access to software that does exactly what people want and 2) there is a very low level of integration between the dozens of pieces of software that we currently use on a daily basis.

And in fact, there really is at least two ways for Semantic Web technologies (and its myriad offshoots, many of them proprietary) to improve the way that we use the Internet.  The first is in fact to provide that "layer" of meaning; making the underlying intent services and content to be made clear to programs and not just developers.  And the second is to actively exploit that layer; building software or carrying out processes intelligently on the behalf of users. 

Traditional software isn't adaptable enough: Mashups and Semantic Web Apps are a better way to do things on the fly  

Need a piece of software to manage the process of planning a wedding and its long list of attendees, suppliers, and dependencies?  How about something to coordinate the delivery of construction materials to a job site for the least total cost including materials and shipping, just in time and in the correct order as the items on the construction schedule are completed?  The possibilities in the consumer and business worlds both are truly endless and reflect that such software can at long lat perhaps fill The Long Tail of IT software demand , which could never cost effectively serve the thousands of mass customized applications that would potentially make using software a dream instead of the chore that it often becomes due to the fact that processes and not just data is what needs to be managed.

And while this -- and by "this" I mean recombinant, self-assembling software that exploits collective intelligence -- is certainly the cutting edge of software development, many companies are beginning to map out this terrain closely and I encourage you to begin tracking them along with me.  Startups and initiatives such as JackBe, Teqlo, OpenKapow, Itensil and a great many others are either wholly or partially enabling the automation of software creation and process management. Interesting, they are usually not via true Semantic Web technology, but by virtue of open, simple, easy-to-describe-and-consume services of the Web 2.0 generation .

This brings us to my last point.  In a panel earlier this year with Adam Bosworth and other notably Web lumuniaries, I responded to an audience question about the difference between Web 2.0 and the Semantic Web by saying "Web 2.0 is what happened while we were waiting for the Semantic Web." And that highlights an interesting point, that this latest generation of tools appears to be built on simple yet proprietary approaches and not on the open but formal Semantic Web technologies.  Whether this points to underlying issue with the usability of Semantic Web 1.0 is hard to say but RSS 1.0 ran into the same issue.  Thus I call this next generation of approaches the "Pragmatic Semantic Web." But I am a bit concerned about the lack of standards and this will be something to watch as we see if this next generation of online software is truly ready to sprout wings and fly. 

What other Web 3.0/Pragmatic Semantic Web companies or projects do you know about?