Research Papers and Videos on Google Bigtable, GFS, Chubby and MapReduce

Thanks all for coming to my talk today on sorted ordered column-family stores at the Silicon Valley Cloud Computing Meetup. Here are the links to the Google’s research papers and a couple of videos that relate to Bigtable (and Google App Engine internals):

Bigtable: A Distributed Storage System for Structured Datahttp://labs.google.com/papers/bigtable.html

The Chubby Lock Service for Loosely-Coupled Distributed Systems

http://labs.google.com/papers/chubby.html

The Google File System

http://labs.google.com/papers/gfs.html

MapReduce: Simplified Data Processing on Large Clusters

http://labs.google.com/papers/mapreduce.html

BigTable: A Distributed Structured Storage System –

http://video.google.com/videoplay?docid=7278544055668715642&hl=en#

Google I/O 2008 – App Engine Datastore Under the Covers –

http://www.youtube.com/watch?v=tx5gdoNpcZM

Enjoy reading the research papers and watching the videos!


Getting Friendly With Document Databases

Here is a draft of what I plan to present in Part 1 of the NoSQL Series on Jan 24th at Fenwick & West in Mountain View, CA (The series has 4 parts in all. It runs between 1/24 and 1/27, everyday at 7pm). The event is  hosted by the Silicon Valley Cloud Computing Meetup.

Topic: Getting Friendly With Document Databases

Scope:
Products covered: MongoDB (mongodb.org) and CouchDB (couchdb.apache.org)
Level: Introductory but not cursory. Full of examples.
Duration: 60 mins. (1 hour) — may have too much for an hour. Could do a bit more than an hour if need be.
Session Contents:
– Document databases
  • What are they?
  • Their essential structure (in the context of MongoDB and CouchDB)
  • Data types supported
  • Schemaless
– Creating, Reading, Updating and Deleting Documents
  • Using MongoDB
  • Using CouchDB
– Querying Documents
  • Filtering
  • Ordering
  • Limiting result set
  • Grouping
  • Joining (?)
(Includes MapReduce)
– Indexes
  • Types
  • How-to
– Very first steps in performance tuning
  • Understanding query plans
  • Faster query results
– A few peculiarities
– Questions
This should give you a head start but an hour isn’t enough to cover all the details so am planning on organizing a follow-up 2 day training in February. See you on Jan 24 at Fenwick & West.

Special Guest at the NIT PowerConnect

The NIT (National Institute of Technology) Almuni Network in the Silicon Valley organizes a monthly power connect networking event. I am honored to be invited as a special guest to their event this evening. I am not an NIT alumni (I attended St. Stephen’s College, XLRI and Courant, NYU) but do know that NIT (which was formerly known as REC) produces a number of very bright engineers every year. Although, IIT is the big global brand from India which has produced a number of very smart and well-recognized engineers, few know that NIT has a lot of great success stories as well.

I will be leading the conversation on NoSQL and Cloud Computing and would participate in a panel discussion on “Snakes and Ladders – How to climb the corporate Ladder” with a bunch of well-known and respected professionals including Bala Sahejpal (IT Director at Juniper Networks), Paul Chen (Director at PapayaMobile), Dilip Saraf (a career coach), Anand Kamannavar (Applied Ventures) and Biren Gandhi (Ex Studio CTO Zynga). Looking forward to the exciting event this evening. If you are coming to the event then look forward to seeing you there.

5 Technology Application Trends in 2011!

5 technology application trends that I think will be most popular in 2011 are:

  1. Tablets, tablets and tablets: iPad started the fever and its not stopping anytime soon. Newer and smarter tablets of all sizes, form factors and capabilities will emerge. Newer and newer applications for these devices will be available.
  2. Big data will get bigger: More and more big data will become available in the public domain and we will see the emergence of newer and smarter storage and analytics solutions in the space. In other words NoSQL and all tools that help manage big data will boom. Cloud will continue its expansion.
  3. Local will be king: Groupon has show the way but there is a lot more to win! Hyperlocal communities will be the way forward. You will see a lot more startups in the space. If you are an investor, don’t forget to put some money there :)
  4. Social networking shakeout & correction: Every boom meets a correction, Facebook and friends will see some correction as well.
  5. Growth of collaboration: the audio-video segment has been largely a consume only space for a while. Collaborative rich communication and interaction will see some innovative new applications.

Happy New Year!

NoSQL Sessions at Silicon Valley Cloud Computing Meetup in January 2011

After Santa Claus has come and gone, NoSQL is coming to town! Come January 2011, I present the core NoSQL ideas, concepts, tools and technologies via a set of 4 day back-to-back sessions at the Silicon Valley Cloud Computing Meetup. The schedule is as follows:

Jan 24th (Monday): NoSQL Series – Part 1: Getting friendly with document databases
Jan 25th (Tuesday): NoSQL Series – Part 2: Nothing beats a distributed hash
Jan 26th (Wednesday): NoSQL Series – Part 3: HBase beyond the “Hello World!”
Jan 27th (Thursday): NoSQL Series – Part 4: Eventually it’s consistent
The venue is:
Fenwick & West
801 California St
Mountain View, CA 94041

Each day session starts at 7pm so you don’t have to miss work to join us for these sessions. Actually, wanted to make sure everyone was tired after a long day’s work so there were less questions :)

Each session is about an hour and a half long with a short break of 5 mins. or so in the middle. There is pizza, veggies and desserts to go along with the talk.

Thanks to Sebastian Stadil for organizing the Silicon Valley Cloud Computing Meetup and making these sessions possible. If you are into big data, cloud computing and web scale stuff and are in the bay area then this meetup is surely the one you should join.

All these talks will leverage my efforts towards writing Wiley’s Professional NoSQL (coming end of Q1/Q2 2011).

Hello BlackBerry PlayBook World!

First Steps with BlackBerry PlayBook AIR SDK

Last week at Adobe MAX 2010, the BlackBerry Playbook was introduced to the world of developers. Positioned as a powerful multi-tasking environment it promises to pose formidable competition in the tablet landscape. The PlayBook OS is built on QNX, the real-time time-tested OS and offers an Adobe AIR based SDK for developers to write their apps.

As soon as I was back from MAX, I was itching to write my first “Hello World” app using the BlackBerry PlayBook SDK. So here in this post is an account of that adventure.

To get started with the PlayBook SDK, first go to the BlackBerry Tablet OS development resources and download the SDK and the simulator for your platform. You will need to get the appropriate versions of the SDK and the simulator for your platform. At the moment, Mac and Windows are supported. I tried my hand with both Mac and Windows. There are “Getting started” guides for Mac and Windows and that’s where my journey began.

To setup the development environment you need the following:

  • The BlackBerry Tablet OS Simulator ISO
  • VMWare Player (or VMWare Fusion on the Mac) to configure and run a virtual environment on the basis of the ISO
  • AIR 2.5 SDK
  • Flash Builder 4.x (the support officially extends to Flash Builder 4.0.1 but you can make it work with FB Burrito)
  • BlackBerry AIR SDK

The PlayBook simulator leverages VMWare Player to create a virtual environment for you to test your apps. When I began to follow the steps in the “Getting started” started guide for Mac and wanted to download and install the VMWare Player on a Mac as per the following guidelines: Configure a virtual machine for the BlackBerry Tablet Simulator, I realized a VMWare Player didn’t exist for a Mac at all. So I downloaded a trail version of VMWare Fusion instead to get things working. Once VMWare Fusion is installed one can follow along the instructions in the “Getting started” guide to setup and configure the simulator. Since that is documented and available online it makes no sense to reproduce it here.

Next, I downloaded the BlackBerry AIR SDK. Flash Builder 4.0.1, AIR 2.5 SDK and Flash Builder Burrito were already installed so I did not need to re-install it. If you don’t have Flash Builder and AIR on your machine then please download and install them before you move forward. Installing the BlackBerry SDK was elementary as a wizard guided through the process. However, one little thing was very odd. The BlackBerry SDK refused to deploy to any Flash Builder other than one named: “Adobe Flash Builder 4″. I have Flash Builder 4 installed as a plug-in and have the standalone FB Burrito that does not have a folder name as stated. So I had to hack around the problem by creating a symbolic link named “Adobe Flash Builder 4″ to my FB Burrito folder on the Mac. On Windows I just configured the SDK, which I will talk about in a bit.

With that done things were working on the Mac.

On the Windows box, I had a different story. Installing the VMWare Player was no problem. It comes for free and installing it requires clicking the installer. That’s it! However, things got tricky right after that. I downloaded the simulator installer but on clicking the installer was greeted with this lack of 64-bit platform support message:

BlackBerry TabletOS Simulator Installer Error on Windows 7

BlackBerry TabletOS Simulator Installer Error on Windows 7

This I think was quite uninviting. Anyway, I worked around this problem. All I needed from this bundle was the TableOS Simulator ISO, so I extracted the installer using 7-zip and traversed down the extracted folder to BlackBerryPlayBookSimulator-Installer-Win\InstallerData\Disk1\InstData, where I again extracted the zipped up file called “Resource1″. Once extracted, I could get the ISO at Resource1\$IA_PROJECT_DIR$\installerdata. From there on I could use the instructions in the “Getting started” guide for Windows to setup the simulator.

As expected, the BlackBerry SDK installer didn’t work on Win 7 either. It again threw up the now familiar message stating lack of Win64 support. Makes me wonder why companies don’t have installers support the 64-bit platform. This is a developer’s tool and not an end user product. Many developers are already using the 64-bit platform and all should and will use it in a couple of years time.

I used the same trick as before and extracted the SDK installer using 7-zip. This time I traversed down the extracted installer to BlackBerryTabletSDK-Air-Installer-0.9.0-Win\InstallerData\Disk1\InstData. There I extracted the “Resource1″ zip file and traversed further down to Resource1\$IA_PROJECT_DIR$\installerdata. In this folder there are two JAR (Java Archive) files that hold the contents of the SDK. The names of these 2 jar files are as follows:

  • blackberry-tablet-sdk-0.9.0_zg_ia_sf.jar
  • qnxsdk_zg_ia_sf.jar

Next, I extracted these two jar files using 7-zip again. After this I created a folder in the “applications” folder, which resides at the root of the C: drive and named it “blackberry-tablet-sdk-0.9.0″, essentially all except the “zg_ia_sf.jar” part of the BlackBerry Tablet jar file name. You can choose any other name, say just blackberry-tablet-sdk or any other. Next, I merged the contents of the two extracted jar files and the Adobe AIR 2.5 SDK (which I assume you would have downloaded and setup by now). That was it! It got my BlackBerry Tablet SDK up and running.

Once the SDK and simulator is setup, you can open up Flash Builder and first configure the BlackBerry Tablet AIR SDK by adding it to the list of “Installed Flex SDKs” as shown in the Figure below:

Installed Flex SDKs

Installed Flex SDKs

Once the setup was complete, I could finally get to writing the “Hello World” app. I wanted the “Hello World” app to be a bit more exciting than printing “Hello World” out to the screen so used the Google Maps Flash API to draw out a map of the area where the Adobe MAX venue, i.e. the LA Convention Center, was.

To create this app, I created a Flex project in Flash Builder and selected it to be an Adobe AIR application. Then I explicitly chose the BlackBerry Tablet AIR SDK as the SDK and finally I chose an ActionScript file as the main file. (As far as I understood, the SDK doesn’t support MXML directly as of now.)

Once the application was written, I compiled the application as usual and then packaged and installed the application to the simulator. I used the command line to package and install the app. You can read about the command line options online. You will need to retrieve the simulator IP to get this to work. Don’t forget to read about retrieving the IP of the simulator.

The command line packager and installer has a format like so:

blackberry-airpackager -package output_bar_file_name -installApp -launchApp project_name-app.xml project_name.swf any_other_project_files -device IP_address

I will not talk about the app itself at the moment but may cover that in a subsequent post, especially once I integrate with a few BlackBerry TabletOS specific features and gestures.

For now I just include a snapshot of the initial “Hello World” version.

Hello BlackBerry PlayBook World!

Hello BlackBerry PlayBook World!

Painlessly Remove All Ruby Gems on Windows

If you want to uninstall all gems from your local Ruby installation on a Linux, Unix or MacOSX box then you can rely on the standard shell commands like “cut” and “xargs”, to make the process easy and effortless. The command itself is a one liner as follows:

gem list | cut -d" " -f1 | xargs gem uninstall -aIx

Read this post titled — Painlessly Remove All Ruby Gems — to learn the details. If you are on Windows though, the usual “cut” and “xargs” are not available. What are the alternatives then? One old school method may be to write a batch file or a script to do the job. However, that method is both a bit clumsy and verbose for a task that could be achieved through a one liner elsewhere. A smarter option then is to use the Windows PowerShell. Lets see how.

First start a PowerShell instance. If you are on Windows 7, it would simply mean typing “PowerShell” (or even “powershell”) on your program search box and then selecting the “Windows PowerShell” program. Once a session is instantiated, type the usual

gem list

command to list all the installed gems. On my machine the output looks like so:

*** LOCAL GEMS ***
abstract (1.0.0)
actionmailer (3.0.0, 3.0.0.beta3, 2.3.5)
actionpack (3.0.0, 3.0.0.beta3, 2.3.5)
activemerchant (1.4.1)
activemodel (3.0.0, 3.0.0.beta3)
activerecord (3.0.0, 3.0.0.beta3, 2.3.5)
activerecord-tableless (0.1.0)
activeresource (3.0.0, 3.0.0.beta3, 2.3.5)
activesupport (3.0.0, 3.0.0.beta3, 2.3.5)
addressable (2.1.1)
arel (1.0.1, 0.3.3)
authlogic (2.1.3)
builder (2.1.2)
bundler (1.0.0, 0.9.25)
buzzr (0.2)
calendar_date_select (1.15)
cgi_multipart_eof_fix (2.5.0)
chronic (0.2.3)
compass (0.8.17)
couchrest (0.37)
crack (0.1.7)
data_objects (0.10.1)
dbd-mysql (0.4.3)
dbi (0.4.3)
deprecated (2.0.1)
dm-core (0.10.2)
do_mysql (0.10.1 x86-mswin32-60)
….

To get a list of all installed gems, one unique entry per line, and containing nothing other than the names one can use the “cut” command on a Unix/Linux/MacOSX machine. With PowerShell though

gem list | cut -d" " -f1

doesn’t work but

gem list | %{$_.split(' ')[0]}

does. The $_ passes the current variable value to the “split” command, which uses a delimiter to split a string. In the example above a space is the delimiter. The parts generated out of the splitting are available as members of an array. Accessing the element at the 0th index of this array returns the first element.

Now that we have the names of all the installed gems, we need to iterate over this list and invoke gem uninstall with flags Iax for each of these. The I is for ignore dependency, a is for all matching gems and x is for no confirmation required. In other words running

gem uninstall -Iax activerecord

should uninstall all gems matching the name — “activerecord” — without any required confirmation.  Using xargs it is easy to pass the current value to a command as one iterates over a list. However, running

gem list | cut -d" " -f1 | xargs gem uninstall -aIx

doesn’t get the job done as xargs is unknown to PowerShell. Don’t be disappointed though for PowerShell has a replacement for xargs and its an elegant one. The position of the $_ makes all the difference. So the one liner that removes all ruby gems is:

gem list | %{$_.split(' ')[0]} | %{gem uninstall -Iax $_ }

Isn’t that nice, simple and just a one liner!

Coast to Coast: New York to California

Over the last many years I was traveling to California for work and conferences and every time I was there I was very attracted to the entrepreneurial spirit and the love for technology lead innovation in the silicon valley. However, all those years I lived in New York City, a place that probably is among the best places in the world to live in — great diversity, vivacity and vibrancy. So despite all desire I would stay in New York and travel everywhere else required. In the last few weeks though, I finally decided to reverse the structure and decided to move physically to the bay area and travel everywhere else. I am still in the process of the move but you will see me more often in the West Coast now.

From a business standpoint, our venture — Treasury of Ideas — will continue to service clients nationwide and around the globe. We will also continue to host New York centric mentoring sessions and conferences/events, like the Flex Camp Wall Street, the next edition of which is coming soon. In the next few weeks we will post our new office addresses in New York and California. The phone numbers and online addresses remain the same. I will be traveling as always and will be in New York every once a while so will get a chance to connect with old friends in New York. Also, would love to hear from friends and readers in the bay area, so drop me a message when you get a chance!

Purpose Maximization

Humans are not horses and it takes more than rewards to get the best out of them. In fact, more rewards for complex tasks that requires sophisticated algorithmic thinking leads to poor performance. If you are confused by now, then watch this brilliant animated presentation:

Given that human tasks are increasingly getting complex and automated intelligent applications is the direction, we are all headed to, does it mean large rigid corporates are going to get drained out of talent? On the corollary, does it mean those who run for money essentially don’t care about challenging work? If autonomy and mastery are of paramount importance, is being an entrepreneur or a freelancer a career all smart people need to choose? If smart agile enterprises that engage employees as people and get the best out of them is the way to go, then why isn’t everyone already doing it? Are companies scared things could go out of hand?

Last but not the least, if people like to contribute effectively where they have the freedom and can enjoy challenges for no monetary reward, only because it allows for purpose maximization then why don’t we use that same energy beyond open source software and free encyclopedias to solve some of the pesky world problems that have been around for a few generations?

What do you think?

Come to Flash and the City 2010 in New York, NY

As I have said earlier, I am on a speaking engagement diet for a few months now and am sticking to a fewer number of conferences than I have in the last few years. The place I appear next is in a conference at my home base — New York City. Thanks to Elad Elrom, we have a great Flash conference, appropriately called — “Flash and the City“, right here in our city. Its a conference that has not only attracted a stunning lineup of speakers (including yours truly, just kidding :)) and includes some of the most relevant topics that would interest any and every Flash developer and architect but is also a conference that promises a lot of fun in the very amazing city. There are city tours, bar hopping events and dinner on the hudson. Doesn’t that sound attractive?

The event is scheduled to take place between May 13 and May 16 at the 3LD Technology Center in downtown Manhattan. You can look-up the schedule to get more details on who speaks on what and when.

My session titled  — “Flash amid the cool and the futuristic web” covers a number of different emerging aspects of the Flash platform that are shaping and will be relevant for the web of the future. So, if you care about the future of the web, do stop by.

See you at Flash and the City!