Unit test for Hive query

Sometimes the soul wants something really extraordinaly... for example, to write a unit test for Hive query :)

Let's how it is possible step be step. So, to write unit test for Hive:

First of all, the local hive instance must be run, and for that we need local metastor (I propose Apache Derby) and directories for temporary data, logs, etc. As all configuration will be read from system properties, I didn't find beter way then set up all of them programaticaly...
Be shure to create all mentioned directories before starting Hive, for example with google Guava:

FileUtils.forceMkdir(HIVE_BASE_DIR);

And after then register all of them in system environment:

        System.setProperty("javax.jdo.option.ConnectionURL", "jdbc:derby:;databaseName=" + HIVE_METADB_DIR.getAbsolutePath() + ";create=true");
        System.setProperty("hive.metastore.warehouse.dir", HIVE_WAREHOUSE_DIR.getAbsolutePath());
        System.setProperty("hive.exec.scratchdir", HIVE_SCRATCH_DIR.getAbsolutePath());
        System.setProperty("hive.exec.local.scratchdir", HIVE_LOCAL_SCRATCH_DIR.getAbsolutePath());
        System.setProperty("hive.metastore.metadb.dir", HIVE_METADB_DIR.getAbsolutePath());
        System.setProperty("test.log.dir", HIVE_LOGS_DIR.getAbsolutePath());
        System.setProperty("hive.querylog.location", HIVE_TMP_DIR.getAbsolutePath());
        System.setProperty("hadoop.tmp.dir", HIVE_HADOOP_TMP_DIR.getAbsolutePath());
        System.setProperty("derby.stream.error.file", HIVE_BASE_DIR.getAbsolutePath() + sep + "derby.log");

After that, the local hive executor might be started:

HiveInterface client = new HiveServer.HiveServerHandler();

In fact, we are ready in this moment. Now I propose to create a Hive table, load data into it and perform some queries. The best practice in Java world is to put all metadata/data for test in separate file, so I put them under resources directory in this example, and here is reading from resource text files:

client.execute(readResourceFile("/Example/table_ddl.hql"));
client.execute("LOAD DATA LOCAL INPATH '" +
                this.getClass().getResource("Example/data.csv").getPath() + "' OVERWRITE  INTO TABLE " + tableName);

Ok, now data in the table and Hive knows about them. Let's perform a query:

client.execute("select sum(revenue), avg(revenue) from " + tableName + " group by state");

Even more, we can register custom function and test it!

client.execute("ADD JAR " + HIVE_BASE_DIR.getAbsolutePath() + jar.getAbsoluteFile());
client.execute("CREATE TEMPORARY FUNCTION TempFun as 'org.my.example.MainFunClass'");

And after that we can call fresh function:

client.execute("select TempFun(revenue) from " + tableName);
String revenueProcessed = client.fetchOne();

Runing Spark Unit Test on Windows 7

It's common situation in enterprises when developers are working on Windows platform. When you are working with Hadoop, it sounds as a f**ing shit, but this is a fact.

Recently, I switched in a favor of Spark instead of traditional MapReduce paradigm and was need to implement some kind of unit/integration testing... of course, it was need to work under Windows 7.

I've written very simple test: run ETL in-memory, without touching Hadoop at all (in future, I'd like to read input from local filesystem):

@Test
def testETL() = {
    val conf = new SparkConf()
    val sc = new SparkContext("local", "test", conf)
    try {
        val etl = new IxtoolsDailyAgg() // empty constructor

        val data = sc.parallelize(List("in1", "in2", "in3"))

        etl.etl(data) // rdd transformation, no access to SparkContext or Hadoop
        Assert.assertTrue(true)
    } finally {
        if(sc != null)
            sc.stop()
    }
}

Bum! I got exception:

java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
 at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318)
 at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333)
 at org.apache.hadoop.util.Shell.<clinit>(Shell.java:326)
 at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
 at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:93)
 at org.apache.hadoop.security.Groups.<init>(Groups.java:77)
 at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:240)
 at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:255)
 at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:283)
 at org.apache.spark.deploy.SparkHadoopUtil.<init>(SparkHadoopUtil.scala:36)
 at org.apache.spark.deploy.SparkHadoopUtil$.<init>(SparkHadoopUtil.scala:109)
 at org.apache.spark.deploy.SparkHadoopUtil$.<clinit>(SparkHadoopUtil.scala)
 at org.apache.spark.SparkContext.<init>(SparkContext.scala:228)
 at org.apache.spark.SparkContext.<init>(SparkContext.scala:97)

What?

org.apache.hadoop.util.Shell.(Shell.java:326)

I swear, I didn't use Hadoop in my code!
Unfortunately, Hadoop configuration is initialized together with SparkContext :( no way to omit it...
I was recommended to install HDP on Windows, but I hate this idea...

I tried the most stupid idea - provide winutils.exe... I hope, it's only the check of environment and Hadoop functionality won't be used if I don't touch it.
So, I downloaded winutils.exe from msdn (msdn still helpful even for hadooper), put it to created directory d:\winutil\bin and then add

System.setProperty("hadoop.home.dir", "d:\\winutil\\") 

at the beginning of my unit test

How to write good unit test for Hadoop MapReduce?

Without a doubt, there is avery common situation when UnitTest (or IntegrationTest) is required to test functionality of MapReduce job. This approach perfect fit TDD, moreover, it gives opportunity to develop MapReduce jobs faster, because there is no needs to redeploy jar on a cluster each time and debugging is easy to use.

The first line of defence is MRUnit. Great framework for unit testing, input/output format independent with possibility to run/test map and reduce functions separately. Unfortunately, this framework has a several meaningful drawbacks. For example, no access to MR counters, or during the MR test only one Mapper allowed.

Local execution mode may be used to overcome MRUnit limitations or create integration test for mapreduce job. Let's assume there is runnable MapReduce tool with several input sources (mappers) and reducer:

public class ExampleMrDriver extends Configured implements Tool {

 public  Job createMRJob(Configuration conf) throws IOException {...}

 @Override
    public int run(String[] strings) throws Exception {
        Configuration conf = getConf();
        Job job = createMRJob(conf);
        return job.waitForCompletion(true) ? 0 : -1;
    }


 public static void main(String[] args) {
        try {
         // run job in a Oozie-friendly manner
            int status = ToolRunner.run(new ExampleMrDriver(), args);
            if(status!=0) {
                System.exit(status);
            }
        } catch (Exception e) {
            e.printStackTrace();
            System.exit(1);
        }
    }

}

Nice integration test (or unit, call and use it as you like) for this Hadoop MapReduce a listed bellow:

private String outputDir;

@BeforeClass
public void createTmpDir() throws IOException {
    outputDir = System.getProperty("java.io.tmpdir"); + "output";
}

@Test
public void test() throws Exception {
    JobConf jobConf = new JobConf();
    jobConf.set("fs.default.name", "file:///"); 
    jobConf.set("mapred.job.tracker", "local"); // local mode
    jobConf.set("mapred.reduce.task", "1"); // only one file is required in output

    // create file w/ input content per mapper in test/resource folder
    jobConf.set("input.dir.2", this.getClass().getResource("/mr/inpu1").getPath());
    jobConf.set("input.dir.1", this.getClass().getResource("/mr/input2").getPath());
    jobConf.set("input.dir.3", this.getClass().getResource("/mr/input3").getPath());
    // expected output will be placed here
    jobConf.set("output.dir", outputDir);

    ExampleMrDriver driver = new ExampleMrDriver();
    driver.setConf(jobConf);
    int exitCode = driver.run(new String[]{});

    Assert.assertEquals(0, exitCode);

    // check content of output file, counters, etc
}

@AfterClass
public void tearDown() throws IOException {
    new File(outputDir).delete();
}

Fix PigUnit issue on Windows

PigUnit is the nice and extremely easy way to test your Pig script. Read more here 
However, it doesn't run on Windows at all. When you write your first PigUnit script, you will get the following exception:

Exception in thread "main" java.io.IOException: Cannot run program "chmod": CreateProcess error=2, The system cannot find the file specified :

In fact, it means Cygwin is not correctly installed. To fix it, you have to download and install Cygwin, after that edit PATH variable and enter the path name to cygwin directory.

Try to run again. The next possible error will be:

ERROR mapReduceLayer.Launcher: Backend error message during job submission java.io.IOException: Failed to set permissions of path: \tmp\hadoop-MyUsername\mapred\staging\MyUsername1049214732.staging to 0700

It means, your temporary directory is not set correctly (or doesn't set at all). Be honest, I tried to set up this temporary directory with the following code:

        pigServer.getPigContext().getProperties().setProperty("pig.temp.dir", "D:/TMP");

        pigServer.getPigContext().getProperties().setProperty("hadoop.tmp.dir", "D:/TMP");

Unfortunately,it doesn't work.... The solution is to set up system property. There are a lot of way to do it, and one of them is to tune java run configuration when you run your test, just add:

 -Djava.io.tmpdir=D:\TMP\

Ok, that's much better, but it's not the finish yet, there is error

java.io.IOException: Failed to set permissions of path: file:/tmp/hadoop-iwonabb/mapred/staging/iwonabb-1931875024/.staging to 0700 

at org.apache.hadoop.fs.RawLocalFileSystem.checkReturnValue(RawLocalFileSystem.java:526) 

that's because of error in the code.

There are several solutions to fix this bug (it is present in Hadoop for a years... :(). One of them, is to use this patch  or fix code and recompile. But for me it was the best way (special, it will be fix only for specefic version, also it is difficult to maintain on several clusters on dev machines and so on).
So, I've decided to change code at runtime with Javassist

So, the solution is a very simple and self-describing:

To apply it, just call from you test before run in. I'd recommend to do it just after PigTest instance creation.

NoSQL unit testing

Definitely, TDD and unit testing are amazing inventions and this is very high recommended to use it during development. That's very usual to have persistence layer in your application. Several years ago it was RDBMS as standard solution. No surprise, it is good idea to test your persistence layer, unit and integration.
So, as result there are a lot of an in-memory databases for unit testing, and there is prefect tool DBUnit for integration testing.
So, when you write integration test you can use DBUnit, which manipulates dataset. It works in the next way:

• before test method: set db content (set state) 
• execute test method
• after test method compare result in db with expected one

What about NoSQL? For example, if you are using MongoDB or HBase, how is it possible to create unit tests without dependencies on standalone server? How is it possible to make integration test with minimum effort to keep data consistency?
I know the answer: NoSQLUnit - NoSQL Unit is a JUnit extension that helps you write NoSQL unit tests. Available on GitHub https://github.com/lordofthejars/nosql-unit

This perfect amazing breathtaking framework provides possibility to create elegant in-memory unit tests and powerful integration tests in DBUnit way! Moreover, it supports a lot of nosql databases:

• MongoDB (not only one instance, but also replica set and sharding!)
• Neo4j
• Cassandra
• HBase
• Redis
• CouchDB
• Infispran engine (never heart about that before)
• ElasticSearch

And what is the most important, documentation is perfect!

So, I want to say THANK YOU VERY MUCH, Alex Soto, you made amazing job!

Do you know any other tools for unit/integration testing for Nosql? Give me a link, please!

Будуємо мурашкою

У цьому дописі акцент робиться на:
1) білді антом після успішного проходження юніт тесту
2) автоматична генерація номера білда/допис дати/тощо з подальшим доступом до цих мета-даних білда з коду

Юніт тести виглядає цілком логічним проганяти перед кожним білдом, тому в антовському build.xml цілком логічно написати щось назразок:

1. <target name="test" depends="compile-test">
2.    <junit failureProperty="tests.failed">
3.    <classpath refid="classpath.libs" />
4.    <classpath>
5.   <pathelement path="build"/>
6.   </classpath>
7.     <formatter type="brief" usefile="false" />
8.     <test name="my.own.TweetProducerTest" />
9.    </junit>
10. </target>

* This source code was highlighted with Source Code Highlighter.

Де compile-test таска для компілу юніт-тестів, а "зміна" tests.failed слугує якраз для відслідковування вдалого проходження тест-кейсу. Лише за умови вдалого проходження тестів таска, що збирає джар:

1. <target name="build" depends="test" unless="tests.failed">
2.   <antcall target="jar"/>
3. </target>

* This source code was highlighted with Source Code Highlighter.

А ось і ця таска, описую також створення маніфесту

1. <target name="jar" depends="compile">
2.     <delete file="tweet.jar"/>
3.     <property name="version.num" value="1.0"/>
4.     <tstamp>
5.       <format property="TODAY" pattern="yyyy-MM-dd HH:mm:ss" />
6.   </tstamp>
7.      <buildnumber file="build.num"/>
8.     <manifest file="MANIFEST.MF">
9.       <attribute name="Built-By" value="${user.name}"/>
10.       <attribute name="Implementation-Version"
11.            value="${version.num}-${build.number}"/>  
12.       <attribute name="Built-Date" value="${TODAY}"/>   
13.   </manifest>
14.     <jar destfile="tweet.jar"
15.       basedir="build"
16.        includes="**/*.class"
17.        excludes="**/*Test*.class"
18.        manifest="MANIFEST.MF"
19.     />
20. </target>

* This source code was highlighted with Source Code Highlighter.

Цей код заслуговує на більш детальний розбір. Почнемо з кінця - створення джару - до нього пакуються усі класс-файли за викюченням тих, імена яких закінчуютсья на Test - це юніт-тести і вони нам не потрібні в нашій джарці.

Загадковий  посилається до відповідного файлу на файловій системі (який модифікувати/видаляти треба обережно) і забезпечує нам номер білду.

Найцікавішою частиною звісно є опис файлу маніфесту, який дозволяє нам задати наші додаткові атрибути. Імена атрибутів цілком описують їх значення... Файл маніфесту вийде приблизно таким:

Manifest-Version: 1.0
Ant-Version: Apache Ant 1.7.1
Created-By: 14.1-b02 (Sun Microsystems Inc.)
Build-By: kostya
Implementation-Version: 1.0-5
Build-Date: 2010-06-12 18:47:34

Для читання цих даних можна використати наступний код (Groovy):

Enumeration e = Thread.currentThread().getContextClassLoader().getResources("META-INF/MANIFEST.MF");
while (e.hasMoreElements()){
JarURLConnection jarConnection = (JarURLConnection)(e.nextElement().openConnection());
Manifest mf = jarConnection.getManifest();
Attributes attr = mf.getMainAttributes();
String libraryFor = attr.getValue("LibraryFor");
attr.getValue("Implementation-Version");

String version = attr.getValue("Implementation-Version");