power of logginig

Just found two amazing tools: Logstash and Kibana

kindly beautiful log visualisation:
http://kibana.org/about.html

that uses logstash (elasticsearch under hood)
http://logstash.net/docs/1.1.5/inputs/file

Logstash from 琛琳 饶

Resolving underscore templates and Java JSTL conflict

Front-end development has made a huge step for the last couple of year... Could you imagine backbone or underscore in earlier 2010? I'm not... now it has a lot of new possibilities, and new changeless for JSTL users.

Today you can create real cool application with using Rest and html/javascript. So, you can (and you have to) separate backend and frontend development. As for me, the real cool thing is a templating engine in javascript. I'm talking about frameworks like underscore.js and mustache.js. Both is awesome, both is cool and must to be used. In this article I'll discuss underscore templatting.

Let's imagine you consider use underscore and read about templates. The Underscore toolkit includes an easy-to-use templating feature that easily integrates with any JSON data source. For example, you need to repeat some fragment of html many times. For Example, you get list of employees in your AJAX request. With template you have to define place for inserting list of users (let's we want to have table).

Your restfull service returns json with name, position and set of phone numbers, you fetch it w/ backbone and create related DOM.

However there is one problem. In JSTL <% %> is using to mark scriplets... So, we have conflict: JSP vs underscore. And JSP wins!  Fortunately, there is solution - you can override underscore template symbols to use <@ @> instead of <% %> for underscore!

    // underscore templating
    $(document).ready(function ()
    {            
     _.templateSettings = {
      interpolate: /\<\@\=(.+?)\@\>/gim,
      evaluate: /\<\@(.+?)\@\>/gim,
      escape: /\<\@\-(.+?)\@\>/gim
  };
     
    });

So, html fragment:

<table id="employees">
</table>

We aim to insert list of employees here with the next information: employee's name, position, phone number(s). Ok, it's really easy. Our template is:

<script id="employee-template" type="text/template">

  <@=name@>
  <@=position@>
  <@ _.each(phones, function(i) {<@=i@>, }@>
 

Template is ready, know just let's init this template:

var template_html = _.template($('#employee-template').html());
    
var h = $(template_html({name: item.get("name"), position: item.get("position"), phones: item.get("phones")}))

$('#employees').append( h );

that's all!

Working with MongoDB using Kundera

When we are talking about JPA2 we usualy imaging relation database and one of ORM (Hibernate or EclipseLink). However, JPA is general approach not only for relation system. It covers NoSQL as well. And one of wonderful JPA implementation oriented on non-relation databases is Kundera.

Initially Kundera was developed for Cassandra. It seems to be a good idea, if you don't use relation you don't have the main problem of different ORM frameworks:) HBase was the next database supported by Kundera, and now is the time for MongoDB

Also, support for relation schema was anonced! Awesome! Be honest, I don't believe the it'll work well. Otherwise we'll get power competitor to Hibernate and K.
But, Kundera is awesome one to creating prototype, when you need base CRUD operations and just to want to try you application with Cassandra, HBase or MongoDB. The beautiful parts is possibility to write JPQL queries instead of native database structures.

Let's figure out how we can use Kundera with MongoDb... Mongo JPA

who has stolen my CPU?

A week ago I bumped into issue w/ AWS CloudWatch. Originally, my load was about 50% and I set CloudWatch to notify me when average CPU utilization > 90% for 5 minutes. And a week ago I got several alert! I was really surprised when recognized that load is more 90% for a long time.
Nothing was changed in application configuration or real load... so, I started investigation. AWS status shows all services operating normally

But the biggest surprise: according to "top" load was less than 10%!
Now I have this load on regular base, for example

(click to view big image)

surprise in Hadoop log

When I started working with Hadoop I was confused by next message in logs:

DEBUG conf.Configuration: java.io.IOException: config(config) 
    at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:225) 
    at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:183) 

It was Hadoop 1.0.3 and I didn't understand "what am I doing wrong?". It was just the newest Hadoop and I didn't find more information in Google, So, I was need to check source code... surprise! Look at it (line 4!):

public Configuration(boolean loadDefaults) {
    this.loadDefaults = loadDefaults;
    if (LOG.isDebugEnabled()) {
      LOG.debug(StringUtils.stringifyException(new IOException("config()")));
    }
    synchronized(Configuration.class) {
      REGISTRY.put(this, null);
    }
    this.storeResource = false;
  }

I can't believe they always log exception... strange way to get stack trace? maybe...

In memorize: REGEXP

The best online java regexp tester http://www.regexplanet.com/advanced/java/index.html

Raspberry Pi cluster for hadoop?

I just thought "Raspberry Pi cluster for hadoop?" Is it possible? does it makes any sense?

Let's think... Hadoop uses hard-drive very-very intensive. Memory.. it's good too have enough memory, but it doesn't critical; I believe 512 MB will be enough. CPU... depends on your code, but usual it's not critical point for map-reduce in general

So, with Raspberry Pi you get (just for $35!):

• RAM 512 MB
• CPU ARM11 700 MHz
• SD with Linux 4-16 GB (you will need to buy it separately)

Some time ago there was the nice article about Paspbery Pi supercomputer: 64 Raspberry Pi computers were connected into the one cluster (via Ethernet); each has 16 GB SD card and it means 1 TB storage for whle cluster (!), and costs about $4000
One concern: access speed to SD card. It isn't good enough and you will need to buy external SSD hard-drive. I assume each Raspberry Pi has to have own SSD (32-64 GB must be enough). So, this solution will be a more expensive that $4000, but cheapest than whole PC or cloud instances.

Let's try to calculate: 64 Raspberry Pi * $35 = 2240, SSD 64 GB * 64 = 4TB costs $4500, whole solution will cost $6500-$7000 for 64 physical node:)

So, does is make sense to build hadoop-oriented cluster? I believe so, what do you think?
At least, it will be a great experiment!

PS. Maybe someone wants to donate money for this experiment? kickstarter sounds resonable here