How do you cross the line?

How do you cross from the online to offline world?

I wrote a post where I mention Gnip as an innovative service, one that can help to increase web services’ scalability on one end and to ease building new mashups applications on the other. Consequently, I was invited by Eric Marcoullier Gnip’s founder and CEO to meet.

I was excited about this meeting but at the same time very confused. I was only blogging for few months then and I had no idea what to expect. I did not know what kind of hat should I wear when I’ll see Eric. On one end I could be a reporter, with a notebook quoting and taking meeting minutes. On the other end I could come to this meeting as a “consultant” (for free) discussing new ideas.  I had no clue.

We met and we talk for an hour but I was still missing the context. I did not took notes and I was probably too nervous because I remember very little from our discussion. I only remember that Eric was very nice and actually have strong roots in MA where I live now. I learned that Gnip is a very young startup but it already did a lot in the short time of its existence. It is changing the way that web services integrate, from the constant, wasteful, chatty data pulling to the scalable and efficient data push. I  learned that the next step for the service/platform  is delivering the actual data along with the event (they are using the publisher subscriber model) then normalizing it for the same entity across multiple web services – a very cool capability. My take: I think that Gnip can become the backbone of the online APIs world. The metaphor is a “smart router“. This time not just for packets but for data and metadata. This “router” can be optimized to distribute data fast, reliably, and to eliminate dups. The service can monitor the health and activity across multiple web services (reporting and dashboard). And maybe even help other web services to recover by storing undo and redo logs. There is always the option to mine the data (across multiple data sources) but I’m not sure how much both sides off the track will be happy about it. The questions? whom to charge the fee from? The producer or the consumer? It is too early to answer these question though.

I probably should have consulted someone before meeting Eric so I could come better prepared to the encounter. I’m sure that Chris Brogan has few good tips on how to deal with this kind of situation.

Anyway, what would you do? I guess that it has to do with your personal/blog’s objectives. But, I’m still working to define mine.

If this meeting would have happened today I would probably come up with a note book or some other means for capturing the content of the meeting. I would probably should’ve think about several key questions to ask and I should have listen more and talk less (always a good advice ). Yet, I’m still dancing between the two options: consulting or reporting?

I feel somehow bad because, at least from my side, I could have done better with this opportunity. Meybe it is like the fist pancake:) I guess that one has to stumble some before acquiring new skills. I’m still grateful for the experience.

So how do you cross to the (off)line? 

What would you expect from a meeting with a founder, CEO, PR rep, other blogger? 

I look  up to the professional bloggers for some insights. For the noobies I can only offer the advice to ask for an advice before you meet.

Picture from Ed Schipul photostream (I think) – thank you.

Outsourcing Data Producer’s open API development and support – a new business opportunity?

So you built a new Web 2.0 like service. It gets some traction and people are crowding in. The site just published an open API. Before you know it, the system crumbles down under the weights of its own success. Sounds familiar?

If your API only exposes “read only” API i.e. an option to pull some of the data out of the system you’re only in half of a trouble. In a case where your API allows the “writing” option too, i.e. modifying system records in the database, now things gets really interesting. Example for read only: Technorati provide blog, bloggers and posts information to Internet Bots and badges. Twitter is an example for both read and write API. Bots built using Twitter API can get members’ status updates as well as automating posting status messages.

Btw, both of te above examples are for Data Producers that are working night and day to scale these open API support.

The problems are generally the same and so does the solutions: performance (throughput and latency)hardware sizing and cost, traffic pattern predictability, load balancing, throttling, caching, stateless web nodes, multi-casting, table partitions (having skilled=$$$ DBA for building high availability database), backup, API format (there are too many of them), message queuing, redundancy, recovery, security, quality of service (premium services), statistics, logging, error handling, monitoring, abuse protection, you name it.

Gnip is a new start-up founded by Eric Marcoullier that is working to address some of these common problems. Reading their blog shows how much thought and sweat is put into addressing some of these common scalability problems. They aim at addressing some the other pain point in the open API arena like consistent API and Identity discovery. Having a consistent/normalized entity ID across multiple web services can solve one of the biggest obstacle today for using WYSIWYG mashup tools like Popfly – but this is for another post:).

If you fit the profile description from the first paragraph reading more about them icould help, but this is not all this post is about.

Let’s assume that the “read” part is getting better due to service like Gnip (ping in reverse) same way that blogging platforms improved new posts indexing using ping service. Now, bots and mashup services don’t need to be “busy waiting” on the API. What about the “write”? What can be done to make this reusable and scalable?

I think that it all come to a new opportunity here to outsource the entire open API development and support, and to save a bundle. Here are some ways to save on this effort through consolidation.

• Reusing hardware through hosting solutions whether physical or virtual like Amazon EC2 or Google App Engine
• Reusing technologies implementation and integration like using memcached, terracotta and many more
• Reusing expertise – Database Administrator and Security experts
• Protocol and meta data standards
• Monitoring tools and technics

Saving: blood, sweat, tears, grief and reputation (in other words avoiding embarrassment).

Bottom line is, that in my opinion, Gnip take it a good distance forward but there is a room for another reusable, consistent and scalable layer between the Data Producer and Gnip.

What do you think?