Storm, distributed and fault-tolerant realtime computation, part 1

Storm is a real big data solution for soft-real time. In the hadoop world batch processing is a main limitation. Let me explain. Imagine, you track some event, however with Hadoop event happened at 11:00AM, it collected, than processed each half hour (in 11:30AM, bacause of Hadoop is batch processing) and this event influences statistics (or whatever) only at 12:00AM. How to do it more "real-time"

Storm comes into play here.

Storm built on idea of streaming data and this is big difference in compare with Hadoop. Not advantage, but difference. Hadoop is a great batch processing system. Data is introduced into the Hadoop file system (by Flume or any other way) and distributed across nodes for processing. Then processing is started, it contains a lot of shuffle operations between nodes and so on, a lot of IO operations. Afterwards,  the result must be picked up from HDFS. 

Storm works with unterminated streams of data. Storm jobs, unlike Hadoop jobs, never stop, instead continuing to process data as it arrives. So, you can think about Strom as about powerful integration framework (like Apache Camel or Spring integration) if you are familiar with it.

So, when even is produced, it is introduced into Storm and go through set of jobs...

Let's look at Storm terminology:

• Tuple is a data structure that represent standard data types (such as ints, floats, and byte arrays) or user-defined types with some additional serialization code (there isn't analog in Java language, but tuples are very usual in Python and special in Erlang);
• Stream is a unterminated data flow, sequence of tuples
• Spout special component which provide data from external source into Storm
• Bolt is a Storm jobs that can be linked in chain and makes any tuples transformation
• Topology structure of spouts and bolt connected by streams (see picture below)

By default, streams are made as ØMQ queues, but it's underhood and you should worry about that (expect cluster installation time).

One features that I explored during development, but I didn't mentioned during reading tutorial: Bolt has to be serializable, so all class members must be serializable too.

Isolation level in RDBMS

In the famous ACID abbreviation, I means Isolation (Isolation level). According to wikipedia, isolation is a property that defines how/when the changes made by one operation become visible to other concurrent operations.

To understand isolations level better, lets review the main colitions that may happend in concurrent
environment:

• lost update - A second transaction writes a second value of a data on top of a first value written by a first concurrent transaction, and the first value is lost to other transactions running concurrently which need, by their precedence, to read the first value.
• dirty read - insert or update data by transaction that will be rollbacked further
• non-repeatable read - first transaction read data, then read them again and the same data is modified
• phantom reads - on the second read in transaction, new data is appearing

So, now that's easy to describe isolation levels with the following table:

Isolation level	JDBC equivalent	Dirty reads	Non-repeatable reads	Phantoms	Lost update
None	TRANSACTION_NONE	may occur	may occur	may occur	may occur
Read Uncommitted	TRANSACTION_READ_UNCOMMITTED	may occur	may occur	may occur	-
Read Committed	TRANSACTION_READ_COMMITTED	-	may occur	may occur	-
Repeatable Read	TRANSACTION_REPEATABLE_READ	-	-	may occur	-
Serializible	TRANSACTION_SERIALIZABLE	-	-	-	-

nginx postaction

Some time ago I thought how is it possible to count downloaded files from resource. I mean, you have a set of files and you need to gather some real-rime statistics on file downloading or just page visiting. I know it's easy to analyze log file. But what if it doesn't enough for you and you want to trigger some another service... with ngnix solution is pretty simple

There is post action functionality that makes ability to call any resource after another resource was called. It is like TRIGGER in regular RDBMS, so simple and self-described example:

Maven assembly plugin fuck up on Spring project making

Yesterday just bumped into Maven assembly plugin issue. I tried to build one-jar project with a log of spring dependencies. It was successful, however I got error on start:

 Configuration problem: Unable to locate Spring NamespaceHandler for XML schema namespace [http://www.springframework.org/schema/context]
Offending resource: class path resource [spring-context.xml]

According to error, I tried to find issue with namespace in spring configuration files... However, it was correct and without issues. But I found spring-related files in my jar which were located in META-INF directory: spring.schemas and spring.handlers, and they have completely strange content - only several namespace, the most of namespaces were lost. So, the problem is with maven assembly plugin. Fortunately, this plugin is enough flexible and gives possibility to manage each step of building.
The correct version (doesn't perform broken merge) is:

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <transformers>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <mainClass>your.main.Class</mainClass>
                                </transformer>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                                    <resource>META-INF/spring.handlers</resource>
                                </transformer>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                                    <resource>META-INF/spring.schemas</resource>
                                </transformer>
                            </transformers>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

NoSQL unit testing

Definitely, TDD and unit testing are amazing inventions and this is very high recommended to use it during development. That's very usual to have persistence layer in your application. Several years ago it was RDBMS as standard solution. No surprise, it is good idea to test your persistence layer, unit and integration.
So, as result there are a lot of an in-memory databases for unit testing, and there is prefect tool DBUnit for integration testing.
So, when you write integration test you can use DBUnit, which manipulates dataset. It works in the next way:

• before test method: set db content (set state) 
• execute test method
• after test method compare result in db with expected one

What about NoSQL? For example, if you are using MongoDB or HBase, how is it possible to create unit tests without dependencies on standalone server? How is it possible to make integration test with minimum effort to keep data consistency?
I know the answer: NoSQLUnit - NoSQL Unit is a JUnit extension that helps you write NoSQL unit tests. Available on GitHub https://github.com/lordofthejars/nosql-unit

This perfect amazing breathtaking framework provides possibility to create elegant in-memory unit tests and powerful integration tests in DBUnit way! Moreover, it supports a lot of nosql databases:

• MongoDB (not only one instance, but also replica set and sharding!)
• Neo4j
• Cassandra
• HBase
• Redis
• CouchDB
• Infispran engine (never heart about that before)
• ElasticSearch

And what is the most important, documentation is perfect!

So, I want to say THANK YOU VERY MUCH, Alex Soto, you made amazing job!

Do you know any other tools for unit/integration testing for Nosql? Give me a link, please!

SQuirreL - a database administration tool

JPA is a popular way to communicate with database in java world, JPA via Hibernata, JPA vi EclipseLink, whatever... however there is instants cumbersome way to test your JPQL queries.
SQuirreL help you! Overview on wiki

When you have a project with JPA and you want to test several jpa queries (aka jpql), you can use SquirreL. Build you project first, then you need to tune SquirreL a bit: go to Squirrel File –> Global Preferences –> Hibernate,
then click Add classpath entity, and add your JDBC driver jar and all project's jars.
Then specify your persistence unit name, and you are ready to run JPQL queries from this tool for your application.
That's much easy than experiments from java code

Moreover, there is a set of useful plugins which helps you to compare databases, create ER-diagrams or execute a specific code on session start

Also, the is possible to query HBase with https://github.com/forcedotcom/phoenix

Two the most important abbreviation

In software design:
S.O.L.I.D.

Single responsibility principle - an object should have only a single responsibility.
Open/closed principle - “software entities … should be open for extension, but closed for modification”.
Liskov substitution principle - “objects in a program should be replaceable with instances of their subtypes without altering the correctness of that program”.
Interface segregation principle - many client-specific interfaces are better than one general-purpose interface
Dependency inversion principle - Depend upon Abstractions. Do not depend upon concretions


In unit testing:
F.I.R.S.T.
Each test must be:
Fast
Independent
Repeatable
Self-Validating (green or red, mo other options)
Timely (write it according to TDD, not in one month, not when you boss requires it)