Spring Data Redis - Retwis-J

RetwisJ web layer is built on top of Spring 3.x MVC framework and JSP as presentation technology. The web tier is intentionally kept to a minimum and simplified as much as possible as it is not the central piece of the application - developers unfamiliar with the two technologies aforementioned should be able to understand the code with minimal effort. It is recommended to use the excellent Spring documentation for more information on Spring MVC.

The entire RetwisJ web front is handled by the RetwisJController, a annotation-based controller that handles the various web requests and model population. Its methods map the actions available to the application: timeline, user mentions, user posts and so forth. The methods of interest are posts and mentions (similar in structure), which handle saving and loading of posts.

The rendering is handled through by JSP pages under WEB-INF/jsp. The bulk of the work, such as listing the followers, the following network, users in commons, posts and so on is handled through the JSP fragments under WEB-INF/templates, namely posts.jspf and network.jspf.

Just like the original Retwis, RetwisJ does not rely on Servlet sessions for user authentication, rather it uses cookie tracking. CookieInterceptor, a Spring MVC interceptor handles that by verifying the cookies of each new request and setting the authentication details. For production environments, we strongly recommend using a dedicated, mature solution such as Spring Security - the sample intent is to showcases the use of Redis in an easy fashion, without introducing other dependencies.

For internationalization (i18n), RetwisJ relies on Spring MVC support, namely ResourceBundleMessageSource and CookieLocaleResolver - see this section in the reference documentation for more information.

The Redis interaction is handled through RetwisRepository class that demos many of the Spring Key Redis classes, such as the atomic counters, collection abstractions, the template but also the SORT/GET pattern for avoid the dreaded n+1 problem.

To interact with the post and user counter, RetwisRepository uses two RedisAtomicLong which, just like java.util.concurrent.AtomicLong classes, allow numbers to be manipulated in an atomic fashion on top of Redis.

Since the users and timeline entries are mapped as lists, they can accessed through the Redis collection abstraction; in this case RedisList interface. This way, items can be loaded and saved through the familiar Collection interface, without having to resort to Redis commands. In fact, the Redis collections are used to interact with the user network and execute operations on it. For example, the code for finding out the common followers of two users looks as follows:

private RedisSet<String> following(String uid) {
   return new DefaultRedisSet<String>(KeyUtils.following(uid), template);
}
      
public Collection<String> commonFollowers(String uid, String targetUid) {
   RedisSet<String> tempSet = following(uid).
       intersectAndStore(following(targetUid),
       KeyUtils.commonFollowers(uid, targetUid));
        
   tempSet.expire(5, TimeUnit.SECONDS);

   return covertUidsToNames(tempSet.getKey());
}

The collections do not hold any date, they simply provide a view into the Redis store. So manipulating data this way, does not trigger any traffic between the application and the store. The method above has intersected two sets (the followers of the given users), stored the result in a dedicated set (with a small timeout to avoid the data lingering around) and then returns it to the user. The result is stored so that it can be properly parsed through convertUidToNames (see below) and to properly paginate through it. A potential improvement would be to check whether the set exists before creating a new one - this way reusing the result of a previous intersection.

A common problem with any store is dealing efficiently with normalized data. In RetwisJ, to avoid data redundancy, the users and the posts are referred to by their ids (uid and pid); for example the keys uid:[uid]:posts contains a list of post ids (pids) not the actual posts. This means that when the posts for a user are loaded, the posts referred in the list need to be loaded. A simple approach would be simply iterate through the list and load each post one by one but clearly this is not efficient as it means a lot of (slow) IO activity between the application and the database.

The best solution in such cases it to use the SORT/GET combination which allows data to be loaded based on its key - more information here. SORT/GET can be seen as the equivalent of RDBMS join. It is particularly handy when loading hashes since it permits field selection avoid loading of unnecessary data. Spring Data provides support for the SORT/GET pattern through its sort method and the SortQuery and BulkMapper interface for querying and mapping the bulk result back to an object. Method convertPidsToPosts shows how these classes can be used load the posts by executing a join over a hash.