Parenting and Enterprise Hadoop Administration


Half awake on a Sunday afternoon writing this entry, I’m remembering the 4am cries coming from my youngest child’s room waking us up today….as well as the 6am wakeup from the older one who is looking for breakfast – and I was the fool to stay up to watch random late-night television as a means for escape.


As a parent, I’ve come to accept that my life (as once known almost 4 years ago) has changed from one of personal achievement to one of family service. Working on the weekends went from a ‘Hey, going to take a few hours out of watching the game’ to now a carefully timed and negotiated scheduling with my wife. There is no time. Period. (Note: For full disclosure on my current ability to write – it is naptime for the youngest right now, my wife is at the gym, it is raining outside – and the eldest is peering over my shoulder as I write this).

Each day flows from man-to-man to zone defense with the kids, depending on the schedule. Once preschool started and the range of activities, play dates,  sports, lessons, doctors appointments, etc. started to grow – My wife and I had to seriously sit down and mark up a calendar. I do travel a decent amount for work, so to make things work, it became necessary to have a single view of what our lives for the month (and beyond) look like. If I get a request to speak at some event, I go immediately to the family calendar and review the schedule before doing anything. All of the family activities and appointments have grown (and keep growing) exponentially. If we did not have some sort of view, some sort of dashboard of all of these activities, we would be at a standstill with execution, with zero efficiency.

Awesome Developer Art!

Awesome Developer Art!

This personal scenario draws a direct parallel to what is needed with big data and open source Hadoop. (Yeah, this is where work part of the discussion takes over). You have vast numbers of smart people that keep building pieces, parts and compilations (…current, stable, partially stable.. not so current) of solutions to support Hadoop based technologies (ala ‘open source’ project). From the never-ending revisions of the core elements to the countless add-on technologies with names such as oozie, jaql, pig, hive, etc. …all with horrible developer driven names and imagery. How the heck can you keep up (as well as even get into the game) when the landscape is always changing?

The concept of ‘Enterprise Hadoop’ solves (at least partially) this issue of the aforementioned mess. I’m not trying to say any of the project is bad – just not very user friendly to get up and running. Similar to my family’s activity calendar there needs to be some sort of central administration here to make Hadoop palatable for the modern enterprise. Companies want to jump right to the analytic based solutions that ride on this big data infrastructure – yet trip up in this technology environment that plays out more like the Wild West.

One item that I would be so bold as to suggest as a mandatory component of an Enterprise Hadoop offering would be some sort of dashboard/calendar of all of things that are going on in the Hadoop environment. At least tie down everything that is going on, and enable administrators with a single view of the truth – so to speak.

IBM BigInsights ConsoleWe built our product, BigInsights, specifically for deploying Hadoop on an enterprise scale. At the core of this initiative is the BigInsights console – a web-based management console (ala my dashboard/calendar idea) for easier administration and real-time views of your big data system.

Administrators use the InfoSphere BigInsights Console to inspect the overall health of the system, as well as complete basic functions such as starting and stopping specific servers and components, and adding nodes to the cluster. Other users can interact with files in the Hadoop Distributed File System (HDFS), manage applications, and work with BigSheets – Which is our own advanced visualization tool for big data (…it comes with BigInsights).

There it is – a single dashboard view into your big data infrastructure.


This is a live screenshot take from my machine

The more I think about it, my own family calendar (…yeah it is hand written) is lacking in a # of these key areas.  I only see basic schedules and when the appointments are – I don’t get to cancel, reschedule or even execute at a glance. IBM’s InfoSphere BigInsights was built from the ground up for Enterprise Hadoop and has built in technologies and features that are meant to not only ease deployment and management of Hadoop – But also reduce the time it takes for you to get your big data analytics up and running.

I on the other hand have to schedule in a nap on a Sunday…

For more information on the IBM BigInsights Console, click here

Add a Comment