Java Microservices: what is the way to go? A Performance Test with Spring Boot, Payara Micro and Grizzly

I want to build a backend service for a personal Android App that I have been working on. I decided to build it in Java. The reasons that led me to this are several and are not the subject of this post. If someone get curious about this, just send me a message and I will be glad to share what were my reasons.

So, the main question that I wanted to figure out was: "What is the best approach in terms of web server and framework to build my microservices API in Java?"

I was looking for something really thin and that implements the JAX-RS 2.0 API (Java official RESTful API specification). This naturally led me to Jersey, since it serves as the JAX-RS reference implementation.

The idea is to build a microservice API, build a Linux Docker container with it, and deploy it in a containerized application service like Google Kubernetes Engine.

So, I went through a lot of articles, documentation, tutorials... and so on. After this, I decided to pick 3 alternatives for testing:

1) Grizzly v2.4 (https://javaee.github.io/grizzly/)

Grizzly is a server application framework build in Java. It implements NIO and is focused in high performance. From the Grizzly web site: 

"Grizzly’s goal is to help developers to build scalable and robust servers using NIO as well as offering extended framework components: Web Framework (HTTP/S), WebSocket, Comet, and more!"

2) Spring Boot and Embedded Tomcat v8.5.31  (https://projects.spring.io/spring-boot/)

Spring is one of the most famous Java frameworks for development of web and enterprise applications, if not the most. It has been used for a long time as an alternative solution to complex and (some times unnecessarily) huge JEE applications. With Spring Boot, it is possible to build a web RESTful API application using Jersey in a fancy and fast way.

3) Payara Micro v5.181 (https://www.payara.fish/payara_micro)

Payara Micro is a 'micro' version of Payara Server. Payara project is a fork from the Glass Fish project, so it is a full JEE implementation.

Now, let's go to the tests!

In order to test all the 3 alternatives, I built a simple mock RESTful application. It does not connect to any other services and provides two simple methods: a method that returns the current list of "customers" (always 10) and a method for registering a new customer. The code for all the 3 alternatives and the loading test script itself (I used Gatling for building and running the tests - https://gatling.io/) can be found in this Github repository:

https://github.com/fabricio-suarte/java-backend-performance-test

Moreover, all implementations use the same JSON lib (Jackson v2.27) for POJO object parsing. All the 3 servers work on the traditional "Worker-thread" blocking IO strategy (one worker-thread per request). Even though Grizzly supports a non blocking IO strategy, the implementation provided by Jersey (and used in this test) does not use it. So, Grizzly assumes its default IO strategy, which is worker-thread.

The tests have been run in my personal laptop, which has the following configuration:

SO
Linux 4.4.0-122-generic #146-Ubuntu SMP Mon Apr 23 15:34:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux.

JVM (Oracle)
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)

CPU
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
CPU(s): 4
Model name: Intel(R) Core(TM) i5-6300HQ CPU @ 2.30GHz
CPU MHz: 799.968
CPU max MHz: 3200,0000
CPU min MHz: 800,0000
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 6144K

RAM
Data Width: 64 bits
Size: 16384 MB
Type: DDR4
Type Detail: Synchronous
Speed: 2133 MHz
Configured Voltage: 1.2 V

Disk
A 480Gb SSD, sata III.

The test simulates a loading of 60,000 unique users / requests in two ways

First, it simulates all the requests in a window of 60 seconds, sending an average of 1000 requests per second to the backend.  Second, I simply changed the script to double this rate, sending the same amount of users / requests to the backend but in a window of 30 seconds (an average of 2000 requests per second).
This way, I had to change some Linux parameters in order to create enough room to such number of simultaneous connections without getting errors like "Cannot assign requested address: localhost/127.0.0.1:8080" and "Request timeout to not-connected after 60000 ms":

Increasing the Linux "soft" limit for file descriptors adding the following line to the /etc/security/limits.conf file (it is necessary to logout and login in order for this change to take effect):

[user]    softy    nofile    100000

Increasing the range of local ephemeral TCP/IP ports and decreasing the timeout for releasing a closed socket (these are not persistent changes):

sysctl net.ipv4.ip_local_port_range="1100 62000"

sysctl net.ipv4.tcp_fin_timeout=5

I ran the test starting one server at time. So, for each one of them, I did:

1. Start the server
2. Run the test script and wait it finishing (first run)
3. Run the test script again, after the server has been properly "warmed" by the first run (second run)

Finally, let's get to the results (response time in milliseconds: less is better)

As it is possible to notice, here are some highlights:

1. There is a huge difference from the first time the test runs to the second. All servers were able to perform much better in the second time the test ran (after properly warmed).
2. The performance for all servers dramatically decrease in the '30 sec.' test.  Although Payara and Grizzly have presented a very low error rate in the 'First Run', Spring Boot + Embedded Tomcat has not. Probably it is just a matter of initial configuration. I did not tweak any server. All severs were started with its own default. I believe that If I had tried to set properly the servers, not only Grizzly and Payara would not have raised any errors but the performance in the 'First Run' would have been better for all servers.
3. Grizzly has presented the best performance, mainly if we consider the '30 sec.' test. In the 'Second Run (30 sec)', it was able to get 9ms less than Spring Boot + Embedded Tomcat and practically half of Payara. Moreover, others statistics (not shown here) are in favor of Grizzly too. I encourage you to take a look at the detailed statistics produced by Gatling (you can find it here, "simulation/results" directory). There are pretty nice percentile graphs and tables. It is also possible to notice that Grizzly got the lowest Standard Deviation, meaning its response time average is really consistent, for instance. 

Conclusion

The purpose of these tests were because I wanted this information for a very particular and personal case / scenario. If you are thinking in use the pieces of information presented here to decide what the way to go (regarding Java Microservices), my advise is: pay attention to your real performance requirements! Do not make a choice based only in what sever has performed better. You must consider other aspects like maintainability, community engagement / support,  availability of documentation and so on. As an example, it was very, very easy to start programing with Spring Boot. Even the Gradle task for generating a single application 'jar' is provided by default. There is a vast documentation available and a huge community of developers using Spring Boot. Regarding Payara Micro, remember that it is a full JEE server, so it can provide a huge variety of Enterprise features in a Microservices fashion. For the great majority of Enterprise applications that I have found in my career, the performance presented by Payara Micro in the tests is enough. So be clever and make your choice wisely.


Some links and references:

Gatling - Performance Testing - http://www.gatling.io/
Project Grizzly  - https://javaee.github.io/grizzly/
Spring Boot - https://projects.spring.io/spring-boot/

JAX-RS Specification - https://github.com/jax-rs/spec

Jersey website - https://jersey.github.io/

Kunernetes Engine - https://cloud.google.com/kubernetes-engine/