Benchmark Setup

Test Scenario

 

Users connect again and again… forever. The full user profile (name, address, phone, etc…) is requested on each connection. This is the preferred mode of testing because it is the most intensive and it is similar to peak load during peak hours.

Bottlenecks can be identified. Unreliable components will break apart:

• Max-out disk, CPU and network (until one becomes the bottleneck)
• Blocking I/O or inefficient I/O will destroy performances
• Logs will saturate disks when not managed properly
• Memory leaks pile up (if present) until the application crashes
• Race condition are more likely to happen and to be detected
• Bad resource usage (e.g. not freeing file and resources) is fatal
• Multiple of those issues can add up fast and kill the application right away

Test Setup

An identity federation requires at least one identity provider and one service provider. They have to be running on two separate machines.

We use the same software to do both roles. It tests both parts of the software and it’s easier to setup.

Software and Configuration

• CentOS 6.7
• OpenDJ 2.6
• OpenAM 12 (tomcat)
• WSO2 Identity Server 5.0 SP1 (tomcat)
• SimpleSAMLphp (apache)
• Shibboleth (tomcat)

All testing is done with 100 000 users. We use OpenDJ to store accounts since it was identified as the fastest LDAP server available.

All IAM applications are configured to provide authentication through various protocols in the same way they would be in production. They are tuned for maximum performances by following the official documentation and performance guides (when there is one available). The middleware (tomcat/JVM, Apache/PHP) are fully configured and tuned for maximum performances as well. We ran performance tests dozen of times to find the best settings for each software.

Hardware

Everything is virtualised on VmWare ESXi servers, unfortunately I can’t disclose our physical hardware setup. Each software runs inside a dedicated VM with the following settings:

4 cores Xeon v3 year 2015
4 GB memory
10 Gbit networking
(shared disks) 16 HDD SAS 10k rpm in RAID
(shared disks) multiple 4GB cache battery-backed raid controllers

Results

Results are given in “iterations/s”. An iteration is a full cycle: login + profile + logout.

A cycle in SAML takes ~ 10 HTTP requests (up to 30 when accounting for intermediate 30X redirects). There isn’t a clear way to give detailed statistics about a protocol like that. It’s easier to look at the full cycle and think of it as a single giant clusterfuck.

OpenID Connect Results

Authentication is done using the “OpenID Connect Password Grant” specification. An application may ask for user attributes when authenticating with this specification, none are given by default. The tests marked as “+profile” explicitly request all the user attributes.

 

SimpleSAMLphp and Shibboleth are at 0 because they don’t support OpenID Connect at all.

SAML Results

Shibboleth is at 0 because I never managed to get it working properly.
SimpleSAMLphp is rounded to 0. See conclusion.

Conclusion

Shibboleth

Actually, I never managed to get Shibboleth working. When I say working I mean the IDP configured on one box, plus the SP on another box, with integration between them and integration with the LDAP server. The documentation is wrong, obsolete, and incomplete. Referring to files that don’t exist, to settings which were removed versions ago. I spent multiple days to get it to work, in vain.

Did I miss something? I don’t know. It cannot be just me, right? I have setup 4 open source IAM and 4 open source LDAP servers in a few weeks, from scratch, fully integrated with each other. I even reversed engineered OpenID Connect and SAML to write JMeter test scenarios. It is proven that I can work around crappy, incomplete or non-existent documentation. In my experience, Shibboleth was missing essential bits to the point of being impossible to setup.

It’s supposed to be running in production for a couple of big projects. I genuinely have no idea how they could do that. Well, actually I do. They either paid Shibboleth to do it for them, they got an older version which is stable or they got a package with preconfigured settings sent by the federation they want to join. Anyway, it doesn’t matter much. Shibboleth would still get a 0 in the first benchmark because it doesn’t support OpenID Connect at all 😀

SimpleSAMLphp

SimpleSAMLphp has terrible performances. It was originally peaking at 2 authentication/s while I was setting up the tests on my laptop (for comparison OpenAM was 55). I tried to debug it and Apache but couldn’t find anything wrong. In the end I decided that it’s not worth testing it, I might as well just round the result to 0. It is slow, plain and simple.

It might be acceptable for a low traffic usage. It can be practical to test other SAML implementations.

WSO2 Identity Server

WSO2 crashes regularly during testing. Users trying to authenticate are rejected, they get a Java stack trace instead of the intended web page. Tomcat has a fail safe to catch unhandled exceptions and it manages to keep the application server running most of the time. The configuration is a endless mess of XML files with hardcoded strings like ‘localhost’ instead of ‘$HOSTNAME’, causing issues at times.

Far from perfection and not reliable enough for my needs. It still works at times though. Given a bit of cleaning, debugging and polishing, it could become a decent second choice in the future.

OpenAM

OpenAM wins every benchmark hands down. It works and it keeps working no matter how much load you give at it. It has the best performances, the best administration tools, the best documentation, the most agents, the most integration options, the most supported protocols. It is superior in every aspect that can be compared.

It’s officially open source but you will have to pay a license per user per year to get security updates, bugfixes, and support. You get what you pay for.

Final Note

I spent a while to test all those things without a clue of what the results would be. I never would have guessed them to be this extreme: So much things failed miserably (except one which is truly immortal), much to my astonishment.

Few people have the resources and time to do and discover all of this by themselves so I decided to make it public for everyone to read and learn.