Lately, I’ve been wondering how much can you improve performance of your microservices just by switching inter-process communication protocol. The default choice of most enetrprise software engineers seems to be JSON REST api and there are strong arguments behind this choice. After all, frontend understands it well, humans understand it well and there is a great tooling around it.

However, have you ever wondered how much money can your organization save on resources, just by switching apis to use e.g. gRPC instead? How many more requests can you handle, just by doing so?

Hold on, we are going to answer this question!

(If you would like to know more on what gRPC is and how it plays with C#, please refer to my previous post.)

Design of the test

So, I wanted to have some concrete numbers on how does gRPC compare to the “mainstream” AspNetCore MVC JSON api. Something that could justify chosing one over the other. The idea is, we’ll make a data structure and implement “echo” API - server, using the same data structure for both the request and the response:

public class TestDto
{
    public string Prop1 { get; set; }
    public string Prop2 { get; set; }
    public string Prop3 { get; set; }
    public int Prop4 { get; set; }
    public int Prop5 { get; set; }
    public int Prop6 { get; set; }
    public double Prop7 { get; set; }
    public double Prop8 { get; set; }
    public double Prop9 { get; set; }
    public bool Prop10 { get; set; }
    public bool Prop11 { get; set; }
    public bool Prop12 { get; set; }
}

Then, we’ll implement two distinct APIs:

  • AspNetCore MVC JSON api
    I chose AspNetCore MVC instead of e.g. NancyFx as it’s the most popular choice in dotnet community at the time. Then, for hosting, I chose Kestrel and also I removed all the logging so that printing to console does not affect results.
    Used versions:
    • .NET Core 2.2
    • Microsoft.AspNetCore.App 2.2.0
  • gRPC
    There, I used the Grpc nuget provided by google and implemented generated base classes, using bindings for self hosting the API.
    Used versions:
    • .NET Core 2.2
    • Google.Protobuf 3.6.1
    • Grpc 1.18.0
    • Grpc.Tools 1.18.0

Both of these APIs have a single method to invoke, which consumes data structure of type TestDto and reply using the same structure. Both request and reply contain GUIDs in the string properties, to prevent any caching that could possibly happen.

(You can see all the code used for this test, and even repeat the experiment itself if you are determined enough, using the sources published on my GitHub.)

How do we measure it?

I also needed some process to actually put a load on the APIs under test. First thought was to use jMeter, but since we want to check how this IPC protocol performs in dotnet envrionment, we should use dotnet client as well.

Therefore, I decided to code something simple myself and quickly came up with a simple little Meter application. It uses HttpClient for doing requests to JSON api and Grpc package for doing requests to gRPC api.

The Meter application will spawn 30 threads and synchronize their start as much as possible, so that each thread makes it’s first request within roughly the same time window. In total, it’s going to do 20 000 requests, one by one, logging results to csv file. But before doing so, it performs a warmup using 2000 requests and 100 threads. This is for JIT (just in time compiler) to catch up and compile all the neccessary code, before we start measuring. Moreover, to measure memory allocation and collect some CPU stack traces, I enabled performance profiler .

Finally, it’d be nice to have some isolated environment, where the test could be performed as we do not want other processes to affect test results and there is too much noise hapenning on my PC.

To mitigate this, I chose two Azure VMs of size B1ms, this is 1 vCPU and 2gb of memory. Not a lot, but we want to measure relative performance there and it did the job just fine. Both machines were running under Debian 9:

  • first one hosted the meter application
  • second une ran system under test.

I wanted to have two machines, so that the meter does not affect api process and having cloud networking involved in the test seemed much closer to the real world use case.

Tell me the numbers!

While performing experiment, I noticed an interesing thing happening. Just after the VM had started and the test was run, I got a standard deviation around 9 in the results. Then, on next attempt it went down to 7 and stayed within this range. The data being presented there is the fastest (on average) of two consecutive trials having the difference in standard devations under 6% ( 5.56% for json, 4.87% grpc).

I don’t want to give a solid judgement there on which protocol is better, as it surely depends on the context and use case. Therefore, I’m gonna leave the interpretation up to you. Let’s look at the numbers then:

Scatter chart

Time the request started on horizontal axis, response time on vertical axis. Point meaning a single response. JSON API Response Scatter gRPC API Response Scatter

Histogram

JSON API Response Histogram gRPC API Response Histogram

Memory statistics

Since I had perfcollect enabled, below you can see GC statistics of both apis. Actually, I was kind of suprised by this big difference of total memory allocated: ~3gb for AspNetCore and only ~400mb for gRPC

JSON API

GC Rollup By Generation, all times are in msec

GenCountMax
Pause
Max
Peak MB
Max Alloc
MB/sec
Total
Pause
Total
Alloc MB
Alloc MB/
MSec GC
Survived MB/
MSec GC
Mean
Pause
Induced
ALL3126.2112.857.821103.62,976.628.73.30
03126.2112.857.821103.62,976.60.03.30
100.00.00.0000.00.00.0NaNNaN0
200.00.00.0000.00.00.0NaNNaN0

gRPC API

GC Rollup By Generation, all times are in msec

GenCountMax
Pause
Max
Peak MB
Max Alloc
MB/sec
Total
Pause
Total
Alloc MB
Alloc MB/
MSec GC
Survived MB/
MSec GC
Mean
Pause
Induced
ALL627.128.175.177130.9382.82.92.10
0487.128.175.17782.5294.90.21.70
1125.111.669.59434.875.30.12.90
227.011.670.24513.612.50.06.80

Explore the data on your own

You can download all the data presented here using the links below. For viewing the traces use perfview tool.