I think that Microsoft, Powerset deal is a great step and accomplishment for both companies.

For Microsoft this acquisition could help to offer something unique in the search arena. They are trying hard to fight their declining market share. This may be the way around Google.

Yet,  it is even better deal for Powerset! In my opinion this is the right exit strategy for someone building, yet another, search engine today. Starting “cheap” by crawling Wikipedia only, apparently, was a very smart move. It is spam free and a data source rich with information. I’m sure that there will be an integration effort due to the different way that Powerset index text, yet, now they will be able to access a larger data source and reuse some of the already built infrastructure. This is aligned with merger as an exist strategy.

Ironically, there is another gain for Microsoft . They will be getting through the Powerset deal some of the former Yahoo’s search expert (like Tim Converse).  I wonder if this recent move by Microsoft will change something in the Yahoo, Microsoft on-going merger deal discussions.

If you want to read more about Powerset and the deal read these two great posts by VentureBeat :

• Microsoft to buy semantic search engine Powerset for $100M plus
• Powerset’s search technology scoop, may scare Google

After 6 month of blogging I decided that I want to continue doing it so I took one small step forward.

I purchased a new domain www.webnomena.com that is now redirected to my blog on WordPress hosting service. I changed the blog’s name from UsingIT to Webnomena (Web Phenomena) because I think that it is more consistent with my writings (you can guess that Webnomenon.com was taken).  I will continue sharing my observations about what that I think is happening in the web world with the focus on search engines, monitoring tools and the blogsphere.

At this time in my life I don’t find the time yet to “decorate” my blog or taking care of the hosting myself. I love to do so and the limitation using WP hosting service are a real drag (no iFrame, no Flash) yet I prefer to spend my time exploring the web and blogging. 

Few things that I learned in the past 6 month:

• About blogging:
  • Blogging is learning - since I started I learned a ton about companies, products and technologies.
  • Blogging is helping me at my work - when looking for a new “skin” for the web app I now expect nothing less than Web 2.0 look and feel (lots of Java Scripts). Netvibe is an inspiration.
  • Blogging is fun - through this blog I had a chance to talk with people all over the world.
  • I can’t blog from the air  - I need to play, explore, interact, develop to come up with new posts.
  • Treat comment like blog post! - The power of a good comment is amazing. A good comment stands out. I spend the same amount of time for writing a single comment as I do for writing blog post.
  • I’m not an artist - I don’t write like some of the great bloggers/writer out there. I just hope that you will find the ideas and information on this blog usefully.
• About searching.
  • We have plenty of new tools to find information - things that it took me long time to find, I later discovered more than once, were saved by lots of other people on del.icio.us. Some of the others you can find in this post I wrote recently: Is there a way around Google?
  • The business world is not really here yet - I think that there is a need for monitoring tools and analytics showing where the right places to deliver your messages are. I think that there is a place for a tool that will help PR and Marketing. A tool that will constantly gather information that can turn into action (actionable). Internet Marketing is not just Web Analytics. I did not see a company/tool that mash it yet. I’m not aware of an enterprise that use a tool like this.
• About the web
  • Empowerment is one of the keys to viral adoption - see blogging, wikis and comments, rating and annotation, twitter following model vs. friends request,  Seesmic, Qik and video streaming from the phone to the web, API/Web services and mashups. Give people the power and they’ll use it!
  • What is the very next things? - let’s see if 6 month is enough to ask these questions.
    • Is it location aware social networks and search technologies? - the kick off is July 11th?
    • Is it Video in addition to photos transmitted from the cell phone to the web? - I was actually never excited about taking pictures from my mobile phone (the rest of the world was). I’m very excited about shooting videos and transmitting them in real-time to the web. It is not cheap yet, work well only using Wi-Fi and not when using the network but same as with blogging it empowers every on of us to become a filed journalist. I think that we will see more Qik like web sites proliferating or integrating with existing social networks.
    • Will Adobe Air take off? - this will help web developers to play on the desktop. I’m curios to know if accompanies with existing products considering this technology for a re-write.  

Now it is your turn…

Some blogs relay more than their content.

Some blogs emits all sort of personality characteristics.

Sometime you can get a hint about it from the tag cloud yet in most cases it is not there to be found. It is beyond the tag cloud. It is only after reading several posts on this blog and comments that you know what kind of experience to expect. It is as if you can predict how the “engagement” will feel like and what kind of “taste” it will leave you with after.

My first examples is TechCrunch. This is an excellent blog about start-up companies and technology .

As you can see from the tag cloud the focus is on product’s and company’s profiles. What that you don’t see/feel is the tone, the ambition, the culture of hard working (the sweat), how competitive this blog is, the “ready to fight” stance, and more.

Yet, after reading this blog for sometime I know what to expect when I go there.

One interesting fact is, and maybe is the key in here, that you get a lot of these feeling not just from the blog post but from the comment section. It is how the communication between the readers and the writer going on that reveals it. If you want to get a quick demonstration go read Surviving the Net by Steve Gillmor  - read the comment section too to see what that I’m talking about.

Maybe if there was a tag cloud for the comments section we could see/feel the blog in its entirety.

My second example is Chris Brogan another excellent blog and blogger.

From the tag cloud you can see the focus on social media promotion for business. You can get a hint about the personality from the howto tag. Chris is willing to share a lot of his knowledge and educate the rest of us about the trade. You can also see that he cares about writing by looking at the writing and the article tags. Yet these are just hints and I looked for them knowing what to expect after reading this blog for a while now.

In this blog, too, the comment section reveals the crowd that is lured to this blog and the communication style. I always leave this blog with “good taste” in my mouth. 

I could provide some more examples but I think that you get the point. Some blogs takes away energy from you while providing important information in return. Some blogs charge you with both energy and valuable information and some blogs are great to “cuddle” with.

I don’t know if there is a way to mark it (personality tag) or search for it. If there is I can only suggest not to avoid looking at the comment section in this kind of analysis. Maybe these guys from Sweden (Jon Kågström and Mattias Östmar from PRfekt) with their attempt to analyze blogger’s personality will find the answer.

For now we can only share this with others using social network tools like Twitter, FriendFeed, Plurk and the rest.

Please share your examples it  in the comment section.

In this blog post I will cover the different ways that entrepreneurs building search engines solutions are positioned to take on the Google challenge.  As you’ll see soon there are many fronts to this battle.

The Challenge

I have to admit that now when I look for something I go Google first. My homepage is Google but mostly I use the Google toolbar. Google is there when I use the web from the browser. Too close. This is my “active search”.In 90% or more of the times I’m satisfied with Google’s results for my “active search”. Most of the time I find what that I need in the very first result page. I use Google as a spell checker, idioms checker, map, business and people finder. I use it when I’m  looking for : images, videos, stock tickers (then I go Yahoo Finance) and more.

When I “listen” to the web I have more options. I do use Google Alerts but not exclusively.  I have Alerts set for some keywords, yes, including my name, blog name and URL but not just those. I have more “ears” on the web like FriendFeed, Twitter, many RSS feeds, Newsletters and more. This is my “passive search“.

So, how can you lure people to use your search solution (so you can sell ads and make money)?

What are the additional dimension/fronts to this battle over content discovery?

Let’s first look at some of the existing fronts:

• Active search
  • Microsoft - new IE install defaults to Live Search as a homepage - failed (around 9%). Cashback?
  • Yahoo - if they were better they did not had to strike a deal with Google but I think that this is as good as it gets (around 20%).
• Building a developers community - Yahoo - the new Search Monkey initiative is empowering and will probably help Yahoo pushing their search as a service while building a developers community. Some claim that they need to open it even more.
• Social bookmaking - not bad but not so great either. It seems better when the search part was a result of the social network content organization and not the original objective. Tags like in del.icio.us can help in finding information with some crowd wisdom. It could short search time too. Yet, it seems better for some categories then others (it is great for developers).
• Sophisticated search algorithm - If aimed at “Active Search” then I’m not sure. If I get 90% of the times what that I want from Google. I may go there after giving up on Google but not as my first choice. Maybe after many times proven better I will consider it but I did not see one that good yet. Do you? I did not play yet with Powerset that uses natural language processing to understand meaning so there is a hope.
• Better UI for the search results - again, If aimed at “Active Search” then this is not the way to take on Google.
• Blog search engine - seems like aiming at the long tail using advertising network is the next attempt taken by Technorati. Probably serving the blogger community and presenting what is hot now on the service’s web site was not the answer (for a business model). Apparently, most people don’t see the difference between a blog and web site when they go looking for information (or even after finding it in a blog). So, they Google (in most cases not using Google blog search engine). I want to see what the creative people at Twingly will come up with. So far, it looks great. What that I see in this front is more opportunities because it is not bound to textual search only. Bloggers, blogs and posts has relationship, i.e. meta data that has value beyond the content. Blogging is discussing, teaching, preaching, mentoring, provoking, guru-ing, sharing. Systems that knows how to capture this meta data, store it properly and leverage it will have a chance to create a advantage over Google in matching ads to blog readers.
• Vertical search - we need to see how this goes but based on this blog post talking about GenieKnows it has some momentum.
• Passive search - using Alerts, Feeds (using pipes) and Filters like Filtrbox - this can keep me away for a while from Google till I need an ad hock search. Same is posting questions - but I found that it depends on the community. Finding what you want could be slow but for tough questions this could be a great option. Yedda is one example.
• Search results aggregation with social touch: I like Xoost’s premise that some searcher are better than others and sharing searches could save time for everybody. The tool searches across multiple search engines - today: Google, Yahoo and Microsoft Live Search.  I’m still trying to figure out all the features in this service (and there are lots of them) but this seems to be a beginning of a long term relationship. Look at some of the feedback after you join in, you can learn a lot from it. One warning, the pages are overloaded with options and that make it hard for first timers to know where to start, so give it some time. Once you get use to it the Channels tie it all together nicely.

Other fronts not covered here:

• Mobile search engines  - I’m yet to explore this domain. I know that Google is investing in this area a lot of effort. Also Powerset did something in this area yet it only bound to Wikipedia for now.
• Social Search engines - there are so many of them (more than 40) - do you see a leader?
• Search focus on products’ discussions - like Omgili -
• Enterprise CMS search engines - not my cup of tea and actually Google is getting there replacing old solutions.

Additional fronts to consider:

• Distance - get in between me and Google. When I launch the browser it is too late for a different search engine today.  Few options:
  • Use Adobe Air for building a light desktop app (that looks like a web page).
  • Integrate with Twhirl or other “distributed discussion” desktop client.
  • Sneak-in through my blog - like what that Zemanta is doing (in a way this is both active and passive).
• Don’t let me leave my “passive search” page/tool/widget. Few options:
  • Maybe by going social.
  • Maybe building a widget that turns my “Active” searches to “Passive” (no matter what engine I use) .

Do you see more fronts to take on or by-pass Google?

Is it a search and destroy….capital mission?

I’m still intrigue by this subject and can’t stop coming up with more questions and thoughts even after I wrote in my previous post. 

I took my blog URL and looked at the results from few URL shortening services.

Origin URL: http://usingit.wordpress.com/

Hare are the result:

http://www.shortesturl.com/?go=e74c2e10f0 - not so useful when you get a longer URL than the original!!
http://tinyurl.com/5j3t8k - most popular
http://www.url.gen.tr/kd - this one is interesting - does it gives you some control over the hash key?
http://snipurl.com/2glld
http://snurl.com/2glnl - use by Twhirl
http://57x.org/05a3
http://is.gd/vX3 - the shortest available today

As you can see the length of the domain name is a key for a short URL. Most services uses around 1-5 characters (A_Z and 0-9) for hashing the long URLs (that should be enough for a while).

Google could use their recently acquired, supper short domain name for shortening URLs: www.g.cn

They are using it for redirecting to the Chinese localized version of their search page. I can only guess that it makes more money than tiny links service.

There is also the shortest domain that I know about: www.com but it does not beat is.gd  - how did they get this domain registered?

If you are looking for information about short domain name look at this very interesting post.

Here is a list of Free Short URL Redirection Services.

Another point: if I’m right then the hyphen (-) is also a valid character in URL names. I’m not sure but I think that the URL cannot start or end with the dash. What is the reason that these services don’t use this extra characters (36 is better than 35)? Is it too complicated because of the start end constraints? Does not worth it? I’m just curious.

Final thoughts: do you think that whatever inside each of these services database is more valuable than what that a crawler can come up with? Think about it:

• These URLs are picked by humans (lots of them in Twitter and Plurk).
• They can keep statistics for how many times each URL was requested.
• They can build a search engine using these links without the need for building a sophisticated crawler for discovering new URLs.
• They can see what is interlinked inside this tiny linksphere

Am I making a big deal out of a tiny subject:)?

Here are some questions about the URL shortening services and methods that I like to share with you. I hope that these are beyond the “how it work?” typical question. If you still want to understand how it works, I found this blog post useful.

You can also find information and to play with URL shortening on the web site of one of the current leading services: TinyURL.com

1. Is there a Google web service for URL compression?

Follow-up question: if not, then how come there is no Google service for URL compression?

2. Do any of the existing URL shrinking services make money?

I would charge Twhirl for it.

3. Is there a way to use Tiny URL for advertising?

Follow-up question:  maybe not so tiny but can a service come up with an Ad URL?

For example: http://gotmilk.com/5%off I know that implementing such service is way, way, way more complicated than the current use of hash function, yet it is more meaningful than

http://tinyurl.com/5j3t8k. 

4. How does search engines deal with tiny URL?

Finding links within Twitter stream for computing page rank and counting blog post’s reactions (”Tiny Link Love”)

5. Do they consider adding to Twhirl a tooltip showing the actual URL? Or something like the SnapShot tool.

6. How much risk do we take putting all our links in one database/hash table?

We can loose lots of links when using services like Twitter and Plurk (services supporting SMS)

7. How soon will we see a built-in function (button) on our wireless phone/blackberry/laptop/browser for url shortening?

Follow-up question: what would be the image on this button?

8. What are the factors that make one URL shortening service better than the rest? Speed, size, robustness, company financial stability, cost, legal terms and  etc..

9. How and will this profoundly going to change things?

Follow-up question:  will we start buying nice Tiny URL name like http://tinyurl.com/R2-D2 (R2-D2). Go, run to find interesting 5 letter words.

10.  Finally, does size matter? Building a huge URL

Follow-up question: Is it funny?

Now, what do you think? Please comment below.

After few month of monitoring Technorati rank progress for few hundreds of blogs I was looking for a way to compare the ones who are making significant positive rank shifts. The way Technorati rank work in is that if the value is going down it means that the blog’s position is improving and there are less blogs ahead of it on the way to the top. It is hard to compare rank moves when the variance in rank is so huge.

So I looked at blogs that had more than 50% positive rank change and I started looking at the speed of their progress.

The way I calculated the (average) speed is:

• Speed = Percentage gain (from baseline) / Duration
• Duration = last rank update date - first rank update date (in whole days)
• The units of this calculation is: percentage change in rank/day (or percent per day).

You can see the full table with the results in here  (using Google Docs  ). :

Blogger’s Speed results - 6/6/2008 (If WP could support (i)Frame it was possible to see this table embaded inside this post, sigh).

Here is a subset of the table:

I highlighted (bolded) few of the lines to show how speedy these bloggers are. There are a couple of bloggers like Doug Haslam and Jennifer Leggio (Mediaphyter) that are showing consistent improvement even if they are not the fastest in their group (they have beautiful positive monotonic rising curve) .

The table is sorted by the starting rank (baseline). The blank lines were added between blocks of bloggers that started in a similar rank range.

Note: I assume that the Technorati ranking system is not liner and somehow things are moving faster where the rank is a really large numbers (i.e. at the bottom of the blogsphere). This make it a little harder to compare between blogs with great disparity in ranks ranges yet it only emphasize how great are the bloggers that manage to move fast even at the top. This is not exact math so please take it with a grain of salt. Maybe one day when somebody come up with a different ranking system he can take my approach in consideration.

A couple of thoughts:

• Speed within category - it will be great to compare how different bloggers are doing within their domain of interest/expertise.
• Acceleration/Deceleration - in this method I actually calculated the average speed moving between the baseline rank and the latest. What that I don’t show here is who’s speed is accelerating and who’s is slowing down. This could be monitored as well. One more thing that I can plot to BlogMon (Twitter)

Since I did not make the links in the table clickable I added them below (you can also click on the link from the Google doc table:

http://travellperkins.com, http://www.multichannelmetrics.com, http://blog.francinekizner.com, http://doughaslam.com, http://www.four20.net, , http://mediaphyter.wordpress.com, http://www.seotops.com, http://www.purplecar.net/, http://www.twitterholics.com, , http://learntoduck.com, http://daisysdeadair.blogspot.com, http://dossy.org/, http://www.prfekt.se, http://gobigalways.com, http://www.socialmediaexplorer.com, http://bing-thegreeninme.blogspot.com, http://sixrevisions.com, http://www.veronicabelmont.com, http://daily.mahalo.com/, http://thenextweb.org, http://www.bloggerbuster.com, http://blog.twitter.com, http://laughingsquid.com, http://www.37signals.com/svn/, http://refueled.net, http://www.blogher.com, http://blogs.abcnews.com/politicalpunch, http://www.designspongeonline.com/, http://blog.makezine.com, http://www.mixx.com

So who do you want to put your bet on?

Only started looking at Xoost - search sharing i.e. United we search, divided we fall (on spam)

I barely had a chance to explore Search Wikia - from Google: “open source Internet search engine, to which the community can contribute.” I call it read/write search. It seems like an interesting concept.

Also, goofing around with Plurk - I’m not sure that I have the time and energy going through another Twitter like service infatuation:). I’m pleased to be able to see the Gantt chart like time-line.

I’m honor by the coming requests to look at new technologies. I wish I had more time to do that.

I’m also waiting for Twingly launch (possibly soon) - check the spam free search - very cool and functional.

RecommendBox is getting more and more interactive. Having more members form my location can make it even more valuable. For some categories it is only useful for a community sharing local knowledge (restaurants, events, places). For others it does not matter so much (books, websites, video games). I can see many areas how this kind of a service improve and I believe it will over time. If this flies they have a chance to make good money from advertisement. People from MA, USA please come and check it out .  It is still in a very early stage of development.

Other things that keep me busy - for one it is work! We have our own release coming soon.

I’m also working on a report that sums some of the progress I made in the “great blogger fishing project” (you can start from this post and keep going down the thread).

My @Blogmon discovered this great resource if you care about SERP: SEOTops, apparently Musa Aykac knows a lot about it and is willing to share (95.7% gain in Technorati rank in less 38 days, WOW! - now: 29898).

Bye for now.

In his post “Could Someone Explain Technorati” Chris Brogan wonders about the consistency, accuracy and reliability of Technorati service. I can’t explain the behavior of the system over there but I can share some of my experience dealing with different challenges using online APIs (web services) and data. The objective here is to help other mashupers to better prepare for future integrations effort across multiple web services. Since it appears that the mashupers community is growing faster than the web service provider I’m sure that more fellow API consumers can share some stories of their own. I will be happy to hear about.

I see three participants perspectives in this “love triangle”: the web site visitor, the mashuper (the API consumer) and the service provider.

My visitor experience:

Chris Brogan talks about his experience from the user perspective in his post. I have nothing to add here but I would say that as a service provider, this should be my top concern satisfying my loyal community. Maybe the way to deal with this in the case from Chris’s post is by monitoring for exceptions (drastic rise or fall in the rank/authority).

My mashup experience:

As I mentioned in some of my earlier posts (here, here and here) I’m working on a small project for finding productive bloggers by monitoring for consistent improvements in their Technorati rank. So on a frequent basis I monitor the rank for over 800 bloggers now. I plot some of the result to a designated Twitter account: blogmon.
The first set of challenge is dealing with volatile data:

• Some times I see no authority in the results (inboundblogs).
• Some times there is no valid last update date in the results: <lastupdate>1970-01-01 00:00:00 GMT</lastupdate>
• Most time there is no author (the user did not add it)
• Some time there are no tags (the user did not add it)
• Some time as Chris mentioned the rank is off for a short period of time

For example see Seth Godin’s Blog rank history:

last update    rank    authority   
2/12/2008    19    8599   
2/25/2008    18    8697   
3/17/2008    19    8658   
3/22/2008    16    8827   
4/10/2008    15    8946   
4/19/2008    16    8882   
4/23/2008     17    8819   
5/1/2008    243       
5/12/2008     17    8828   
5/14/2008     16    8863   
5/20/2008    15    8890   

These are the details that a consumer of online volatile data must plan and look for ways to compensate for.

Some suggestions:

• Check the validity of the date
• Don’t just count on the last result i.e. search for the last valid result and monitor over time.
• Be prepare to plot partial results (e.g. no top tags or author).
• Most important: guard your data i.e. protect what that you take from the service  and store in your records.

 
The next set of challenge has to do with the web service behavior:

• I get the fowling error once or twice: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
• Some API requests come back with:

<HTML>
     <HEAD>
          <META HTTP-EQUIV=”REFRESH” CONTENT=”2; URL=http://api.technorati.com/bloginfo?url=****&key=****&version=0.9&start=1&limit=30&claim=0&highlight=0″>
      <TITLE>****</TITLE>
      </HEAD>
      <BODY>
           <CENTER>
               <IMG SRC=”***”>
               <BR>
           </CENTER>
     </BODY>
</HTML>

**I intentionally masked the URL, title, image and my developer Key with ****

This result can crash your system if not handled.

• Finally: and I get this one a lot

<?xml … “http://api.technorati.com/dtd/tapi-002.xml”>
<tapi version=”1.0″>
    <document>
        <result>
            <error>You have used up your daily allotment of Technorati API queries.</error>
        </result>
    </document>
</tapi>

 Some suggestions:

• I can’t picture my dev world without Exception Handling - this is the ultimate protection against web service unexpected behavior in this specific case. So guard any call, loading XML result and data parsing by wrapping them with a try and catch block.
• Logging - log expected and unexpected behavior for later analysis and recovery.
• Build the system so exceptions are caught, logged but the execution can move on to the next task.
• This is something that I learned from a smart Army office: “If there is a doubt there is no doubt” basically saying that it is better to not report at all than to report inaccurate data.
• Find ways to minimize the API calls  - e.g. I ask for tags only when I find a blog worth reporting on
• A thought: I’m not an expert in XML and DTD but could it be that using DTD slows down the web service. If you know more about it please share with me/us. Is this really necessary on a read only calls?

About the service:

I can’t talk much about what that a web service provider feels or experience (I’m sure that Ian Kallen from Technorati has a lot to share about this subject) but I want to say few things:

• Please don’t get this post wrong I’m a fan of Technorati - I use it and deeply  appreciate their service and thankful for having the option using the APIs . As I said earlier the intention is to share from experience and to allow you to better prepare for such effort.
• I guess that it is hard to estimate the load on the system with such growth in the number of mashupers out there. So my heart is with them.
• There are two more threats that the web service provider needs to protect itself from and I’m sure that those consume some energy:  protecting the hard gather data and its environments from abuse and malicious attacks.

One last comment: ironically I had none problems with Twitter so far:)  but I’m aware of the pain that some of the Twitter API user suffer occasionally.

Things are going real-time. Social network, decentralize me, or semantic web are all great and we will see more of those coming but the most disruptive change that is already happening is that the web brings us all tougher in real-time. It allows us to read/write the web at any time from anywhere.

There are few catalyzers for this change. The first is the mobile technology including the devices, integrated GPS and the bandwidth, the second is that we are getting use to be always connected, and the third is all the good stuff that is happening on the web.

Now, I’m not talking about the future. I’m talking about the here, now and more just a little later.

Some examples:

Twitter made the connection between SMS and the web like no other tool before. There is nothing more real-time than Twitter. News, gossip, agenda, help, you name it.

Nokia N95 + Qik - real-time video streaming

BriteKite - Location-based social networking - in real-time.

What stopping us?

I think that the biggest obstacle for mass adoption, at this point, is the cost of the data plan. If that cost will go down we can all enjoy:

Read:  mobile constant feeds aggregation getting location, timely, and profiled based information.

Write: sharing our current experience using text, audio and video.

So what will happen a little later?

• Creating real-time social events
• Getting recommendation as we approach a new location that are tailored to our profile
• Knowing who’s in the area (that could be annoying too - it requires some configuration about who you want to know that you are around).
• Car pooling
• Checking for bargains in the area (pool - not push)
• Possible: Instant check-out - just go through the door with the purchased items and check the bill(one at a time)  - combination of RFID and the mobile device (beware if someone steal your phone). 
• Mobile Peer-to-Peer - sharing the information among us off the server

Can you see more ways that WebRT can enrich your experience in real-time?

Update 5-12-08: I guess that I my timing was good RIM Introduced today the BlackBerry Bold Smartphone including GPS, Wi-Fi® support and Video camera.