How to write good unit test for Hadoop MapReduce?

Without a doubt, there is avery common situation when UnitTest (or IntegrationTest) is required to test functionality of MapReduce job. This approach perfect fit TDD, moreover, it gives opportunity to develop MapReduce jobs faster, because there is no needs to redeploy jar on a cluster each time and debugging is easy to use.

The first line of defence is MRUnit. Great framework for unit testing, input/output format independent with possibility to run/test map and reduce functions separately. Unfortunately, this framework has a several meaningful drawbacks. For example, no access to MR counters, or during the MR test only one Mapper allowed.

Local execution mode may be used to overcome MRUnit limitations or create integration test for mapreduce job. Let's assume there is runnable MapReduce tool with several input sources (mappers) and reducer:

public class ExampleMrDriver extends Configured implements Tool {

 public  Job createMRJob(Configuration conf) throws IOException {...}

 @Override
    public int run(String[] strings) throws Exception {
        Configuration conf = getConf();
        Job job = createMRJob(conf);
        return job.waitForCompletion(true) ? 0 : -1;
    }


 public static void main(String[] args) {
        try {
         // run job in a Oozie-friendly manner
            int status = ToolRunner.run(new ExampleMrDriver(), args);
            if(status!=0) {
                System.exit(status);
            }
        } catch (Exception e) {
            e.printStackTrace();
            System.exit(1);
        }
    }

}

Nice integration test (or unit, call and use it as you like) for this Hadoop MapReduce a listed bellow:

private String outputDir;

@BeforeClass
public void createTmpDir() throws IOException {
    outputDir = System.getProperty("java.io.tmpdir"); + "output";
}

@Test
public void test() throws Exception {
    JobConf jobConf = new JobConf();
    jobConf.set("fs.default.name", "file:///"); 
    jobConf.set("mapred.job.tracker", "local"); // local mode
    jobConf.set("mapred.reduce.task", "1"); // only one file is required in output

    // create file w/ input content per mapper in test/resource folder
    jobConf.set("input.dir.2", this.getClass().getResource("/mr/inpu1").getPath());
    jobConf.set("input.dir.1", this.getClass().getResource("/mr/input2").getPath());
    jobConf.set("input.dir.3", this.getClass().getResource("/mr/input3").getPath());
    // expected output will be placed here
    jobConf.set("output.dir", outputDir);

    ExampleMrDriver driver = new ExampleMrDriver();
    driver.setConf(jobConf);
    int exitCode = driver.run(new String[]{});

    Assert.assertEquals(0, exitCode);

    // check content of output file, counters, etc
}

@AfterClass
public void tearDown() throws IOException {
    new File(outputDir).delete();
}

NoSQL unit testing

Definitely, TDD and unit testing are amazing inventions and this is very high recommended to use it during development. That's very usual to have persistence layer in your application. Several years ago it was RDBMS as standard solution. No surprise, it is good idea to test your persistence layer, unit and integration.
So, as result there are a lot of an in-memory databases for unit testing, and there is prefect tool DBUnit for integration testing.
So, when you write integration test you can use DBUnit, which manipulates dataset. It works in the next way:

• before test method: set db content (set state) 
• execute test method
• after test method compare result in db with expected one

What about NoSQL? For example, if you are using MongoDB or HBase, how is it possible to create unit tests without dependencies on standalone server? How is it possible to make integration test with minimum effort to keep data consistency?
I know the answer: NoSQLUnit - NoSQL Unit is a JUnit extension that helps you write NoSQL unit tests. Available on GitHub https://github.com/lordofthejars/nosql-unit

This perfect amazing breathtaking framework provides possibility to create elegant in-memory unit tests and powerful integration tests in DBUnit way! Moreover, it supports a lot of nosql databases:

• MongoDB (not only one instance, but also replica set and sharding!)
• Neo4j
• Cassandra
• HBase
• Redis
• CouchDB
• Infispran engine (never heart about that before)
• ElasticSearch

And what is the most important, documentation is perfect!

So, I want to say THANK YOU VERY MUCH, Alex Soto, you made amazing job!

Do you know any other tools for unit/integration testing for Nosql? Give me a link, please!

funny trap : JUnit vs TestNG

I've found a funny trap just several minutes ago:) If you have strong background in JUnit and then you start coding in TestNG, you can fall in trouble with asserts.
In JUnit we have: assert(expected, actual)
In TestNG: assert(actual, expected)
So, be careful!