Studying the impact of innovation on business and society

The Human Algorithm: How Google Ranks Tweets in Real-Time Search

In 2009, Google struck a deal with Twitter, rumored at $15 million, to integrate tweets into keyword related Google searches. And last month, Google also integrated real-time search technology to surface blog posts and news content as they hit the Web – dramatically improving the previous five to 15 minutes its spiders would take to crawl the Web. I should also note that Collecta also offers the ability to search the real-time Web, but its results also include popular networks within the social Web. Between Google and Collecta, Twitter Search is starting to show its age.

The opportunities and benefits of accessing the real-time Web also represent its most notable deficiencies – the ability to truly focus the stream of cascading information into a river of relevance. Companies such as My6Sense are using a form of “digital intuition” to escalate tweets that match our patterns, behavior, and content we read.

We are now staring in the face of a more sophisticated era of real-time search that will further advance, localize and personalize over time. And, everything starts with the experimentation of sophisticated algorithms that filter and rank the content we’re hoping to discover.

For example, Google recently adapted its PageRank technology for surfacing related tweets. PageRank was originally developed to help find relevant Web pages through traditional search and was Google’s primary differentiation in a world of commodity search platforms. Essentially, Google’s PageRank assesses the importance of Web pages tied to keywords based on link structure. Authority is determined by the quantity and quality of inbound links to each page as well as the branches of outlying link relationships that tie other pages to those within the first degree of inbound connections. In other words, the more links to a page and the more linkers to each link, the greater the value of the original page.

The challenge with real-time search is tying tweets or other social content to notable producers and their networks of reputed followers and sub-follower architectures.

In an interview with Technology Review, Amit Singhal, a Google Fellow who led development of real-time search, said “You earn reputation, and then you give reputation. If lots of people follow you, and then you follow someone–then even though this [new person] does not have lots of followers, his tweet is deemed valuable because his followers are themselves followed widely.

As Singhal emphasized, “It is definitely, definitely more than a popularity contest.”

Google also examines the signal in the noise, to surface the most relevant tweets related to common as well as obscure subjects. And as Twitter itself advances the technology that packages tweets, such as geo-location data, we can expect to see a rapid evolution of real-time search.

Basically, a follower is the equivalent of one page linking to another on the Web. Google recognizes each as a form of recommendation. So as higher quality pages link to sources, the original page increases in value. In the Social Web, reputed users who follow other users inherently increase the stature of the individual to whom they connect.

Searching for a particular keyword now will produce qualified results for Web pages and also content published in Twitter and other social networks, ranked by the authority of the page and publisher of social objects as assessed by PageRank technology.

In the eyes of Google, the adaptation of PageRank for Social Media essentially creates a human algorithm or a PeopleRank of sorts that may eventually serve as a foundation for also assessing the authority of an individual in the social Web.

Other companies are also introducing new services to measure general authority for individuals online. Klout, for example, developed a sophisticated platform for measuring the influence of users in Twitter. Based on three sophisticated stages of semantic calculation (True Reach, Amplification Probability, Network Value) Klout can determine not only the level of influence of any user on Twitter but also the most influential voices tied to topics or keywords. Microsoft’s search engine, Bing, is also including tweets in its real-time search feed and could, for instance, integrate Klout’s influence engine to rank tweets and other social objects to qualify results.

But while the idea of ranking influence on the social web is interesting and necessary, it is far from perfected. Running searches in either engine today will only reinforce this sentiment. However, with that said, it is helping us by reducing the obstacles that typically prevent or prolong the process of finding pertinent information. It will only improve over time regardless of our personal views on establishing a hierarchy of people in social media.

As the human algorithm continues to evolve, it transforms the definition of and logic for relationships. We’re adapting how we connect to one another and also constructing new roads for sharing, filtering, and ranking relevant social objects. The ties that bind us now serve as the source of how we discover information and also how it finds us. And as such, the relationships we maintain on the Social Web determine the ranking of the content we produce and its place within the social hierarchy of search results.

Perhaps the next iterations of Search Engine Optimization (SEO) and Social Media Optimization (SMO) will focus on enhancing the link structures of human relationships to escalate the prominence of our stature and the social objects we create and share.

Connect with Brian Solis: Twitter, LinkedIn, Tumblr, Facebook

Pre-order the next book, Engage!



Click the image below to get the current book, poster, or iPhone app:



Image Credit: Shutterstock

142 COMMENTS ON THIS POST To “The Human Algorithm: How Google Ranks Tweets in Real-Time Search”

  1. julian says:

    hello this a test

  2. dougmcisaac says:

    Hi Brian,

    Thanks for sharing your insights. Ive been anxiously studying what happens in the social search / “friendrank” arena for over a year and the addition of Twitter, Facebook and MySpace results in search is changing the landscape and giving SEO experts fits trying to figure out how to optimize for it. As the algorithms are improved I think the SEO industry is going to have to turn to producing content that people actually want to read and share instead of gaming the system.

    I also noticed that Google is testing “Results from people in your social circle” with – “My social circle – My social content” This is something that we knew was coming, but it's always interesting to start seeing Google actually start testing. The results have been interesting so far. Have you seen those yet?

    Doug

  3. willflavell says:

    Does anyone still use Twitter Search?

    But seriously, great post Brian. It is always a pleasure to read your work. Cool to see PageRank applied to humans. Does anyone have info on how Bing ranks their tweets?

    Best,
    Will

  4. twitter used as a benchmark to measure the relevancy of a website?
    according to the public goal of the site sounds problematic, such as a website whose audience is a country that does not use a lot of twitter, or an age in which social networks are not attractive.

    excellent post

  5. David Reske says:

    Interested in seeing where this goes and the solutions it brings. Thanks for the great post.

  6. John McTigue says:

    I'm wondering if the word “optimization” is becoming obsolete. If our conversations are all to be overheard and indexed, why should we worry about keywords and placement? Maybe we should focus on the conversations themselves and not try to “move up” in the system.

  7. Hi Brian.

    Clay Shirky has discussed this issue, on what he calls “Algorithmic Authority”, addressing also the problem on how to value reputation.
    With real-time search, each individual must be interpreted by Google and i find it hard to extract reputation without context. Even if many people are linking to something, that doesn't necessarily means it's important to my search query.

    Social search will help figuring out what's important, but in the end, as usual with Google and the whole Web 2.0, the more people use it, the more perfected it will get.

  8. Simon U Ford says:

    Brian,

    Social or friend rank algorithm runs a lot deeper than social links from a users profile page, to profile page. They have done for some time now. If you search the term “Google friend connect” in YouTube. You will see a video I published over a year ago (publisher was EventsListed). It still sits at number 2 ahead of 10+ Google owned videos about their own product.

    At the time my video ranked number 1 in Youtube. I had a second video called “Beating Google Friend Connect” showcasing just how I beat Google sitting in the 2nd spot as well. That's positions 1 and 2 ahead of more than 10 Google developer videos for months and months. This was shortly after Google launched their Google friend connect widget. Some of Google's videos had 200,000+ views and any one of Googles videos had many more comments than both my videos combined.

    My YouTube video page therefore held out 4th position in the Google.com search engine as well. This is behind 3 Google owned properties ahead of every A list website and blog on the open web. This is at a time when Google Friend Connect was the talk of the town.

    The reason I ranked so highly came down to a few factors.

    1. The number of comments my video's received in the first 24 hours compared to Googles videos.
    2. What was even more influential was the percentage of the total video (in minutes) that my commenter's (voters) actually sat through, compared to the percentage of the total video Google's commenter's watched.

    3. Especially the comparative statistics between commenter's who watched and commented on both videos. I specifically asked my audience to watch my videos from start to finish and then comment within 24 hours of publishing. Then I told them to watch Googles video from start to 30 seconds in, then stop it mid video and then comment on the Google video as well. This provided YouTubes algorithm enough comparative data between common commenter's which I believe to have been the difference (having plotted / compared all of the other variables).

    This is why the social (friend) rank system beats page rank hands down. Beating page rank only requires the right knowledge to game it. Beating the friend or social rank system requires both the knowledge and having a very responsive, large audience of followers who listen to you before you would even try.

    • Hmmm interesting… I ran a campaign w/300 vids and noticed the comments correlation. I also felt that the number of subscribers to the feed was a signal/factor in ranking and why IMO, not having vids on YouTube is a huge mistake if you are trying to optimize for Video slots in Google Universal SERPs. Google has access to all the data through YouTube so… these are very hard signals to spam.

    • It's interesting to know this fact and even I use twitter more and I know that google index twitter frequently.

  9. netaffinity says:

    Hi Brian,

    Love the article, topical and on the button as always. There is a real and definite shift in how 'search' is dealing with the real-time web. Google and Bing are struggling to present it in a real and meaningful way. I think there are great opportunities for new entrants to the search space and love the collecta concept.

    Keep up the good work.

    Will.

  10. What is interesting is Plurk fumbled and never made it with Twitter, yet had plurk allowed dofollow links would it have taken off and competed with Twitter?

  11. Chris says:

    It will be very interesting to see how this evolves. I wonder what or who will be used as baseline to judge everyone else. How will influence will be tied to each of the social networks. How will we be rewarded or penalized based on the networks we use or don't use. How will friending/following change if we know it will impact our status. There certainly is a lot to come of this.

    -Chris

  12. cyuskoff says:

    Interested to see how this will impact video searches so they are more effective, particularly with realtime video streaming becoming all the rage.

  13. Brian,

    Thank you for the introduction to Klout. I was surprised at my result. After thinking about it though, it makes sense. This is a great way to REALLY see what your influence on Twitter is, rather than what you think or hope that it is.

    Much appreciated, and I always enjoy your blog posts. I can tell you spend time and energy on them.

    Jeff

  14. Great post thanks Brian! I interviewed Joe Fernandez recently on my blog if you and your readers are interested 🙂 http://rich-baker.com/2010/01/15/klouts-ceo-tal

  15. I really dont think ranking tweets is such a good idea. I mean can you imaginge the first page of gogle! I personally couldnt give a crap what someone does on a day to day basis, keep it at twitter or facebook, dont be ruining the search engine. Goddamit.

  16. John S says:

    That is truly fascinating. I wonder if people will be able to game the system. For instance, will a person who is followed by 20,000 people and follows 20 have the same rank as someone who is followed by 25,000 people but is following 90,000? It will be interesting to note how page ranks start changing over time as this new system is implemented.

  17. Fin says:

    Was also very interested to read about Klout. Thanks for yet another great post Brian.

  18. rachelleh says:

    Interesting – thanks!

  19. If you involve yourself in developing search engine algorithms and keep up with.Google's real-time search feature posts updates as things happen around the web.

  20. Pingback: F.A.Z.-Community
  21. Micro Blogging should have its own platform to be indexed/crawled by Google's spiders so it's not just exclusively designed for twitter.

  22. MrJorin says:

    Only want to say your article is awesome. The clarity in your post is simply striking

    and i can take for granted you are an expert on this field. Well with your permission

    allow me to grab your rss feed to keep up to date with forthcoming post. Thanks a

    million and please keep up the solid work.

  23. Though, Google seems to be indexing new sites a lot harder, slower than 10 years ago. I know, the internet is full and there are lots of spammers. They also have trouble finding some content and backlinks out there…

Leave a Reply

Your email address will not be published. Required fields are marked *

Join Our Mailing List

You have Successfully Subscribed!

Stay Connected