Security scanning services are great to increase awareness for two reasons. First of all, they allow you to easily evaluate the security of your own application, and give you a very specific path towards improving your security. The second reason is that these services allow anyone to check how well you’re doing, and call you out if you’re not delivering your A game, which is exactly what happened to the Belgian banks last year.
There is a portion of website owners out there that knows security is important, and really benefits from these services to help them improve their setup. Unfortunately, the majority of websites is lagging behind, essentially leaving their users vulnerable. In these cases, the only thing that may help them fix their security is the potential reputation damage that comes from having a bad score on one of these tests.
This essentially means that the people behind these scanning service have quite a big responsibility. They have to decide which settings are considered secure, and which are considered to be insecure. They have to decide how much these positive or negative results influence the score, and they probably have to deal with a lot of criticism by people that have another opinion. As we go in more detail below, it will become obvious that these decisions are far from trivial.
SSL Server Test
The SSL Server Test, driven by Ivan Ristić of Qualys SSL Labs, takes your HTTPS deployment under scrutiny, and grants you a score between F and A+. The test investigates an impressive list of security properties of HTTPS, including the validity of the certificate chain, the supported protocol versions, the available cipher suites, and whether you’re vulnerable to several well-known attacks.
Two example ratings from opposite sides of the spectrum from the SSL Server Test
The rating system behind the test is fairly well-documented, and consists of a category-based score, which eventually results in a final percentage, that can be converted into a letter grade. There are a few all-or-nothing rules that directly result in a lower score, such as support for the old SSLv2 and SSLv3.
The decisions that need to be made to evaluate an HTTPS deployment are generally not very ambiguous. A server either supports SSLv3 or not, where the former is considered insecure, and the latter is not. However, some decisions are more difficult. For example, the test used to penalize services that did not mitigate the BEAST attack, but reversed this decision once most clients implemented mitigations against this attack. Clearly a decision with a lot more to it than a yes/no answer.
This kind of decisions becomes even more difficult once you move from the network level to the application level, as securityheaders.io does.
Application-level web security has definitely changed a lot over the years, and numerous new security features have been added to the browser. These features are known as server-driven browser-enforced security policies: a policy defined by the server, tailored towards the application, and enforced by the browser. The HttpOnly and Secure cookie flags, which tell the browser how to restrict the use of certain cookies, are a simple example of such a policy. A complex example is the Content-Security-Policy header, which enables fine-grained control over the locations to include resources from, and where outgoing requests can be submitted to.
Securityheaders.io, created and maintained by Scott Helme, investigates various of these security features that are enabled through a response header. Based on how many features that are enabled, you get a score between F and A+. An F means that you have not enabled any security features, while an A+ means that you have enabled all 6 security features that the test currently checks for.
Compared to the SSL Server Test, the decisions that need to be made by securityheaders.io are a lot more ambiguous. How do you determine which setting for a security header is considered to be secure? The decisions range from straightforward to extremely complex. Let’s look at a few examples below.
Two straightforward security headers to check for are Strict-Transport-Security and X-Content-Type-Options. The former comes with a max-age value which needs to be high enough, and the latter only has one valid value, nosniff. For these headers, the decision is easy to make.
Three other headers are already a lot more ambiguous: Public-Key-Pins, X-Frame-Options and X-XSS-Protection. Without going into too much details on these headers, they allow a lot of freedom in their values, and the strictness of their protection is highly dependent on the application in question. For example, the X-Frame-Options header can be configured with the following values: DENY, SAME-ORIGIN, ALLOW-FROM and ALLOWALL. The first three settings restrict the current browser behavior of allowing framing by anyone, so they increase the security. The latter allows framing by anyone, which is essentially the same as the default browser behavior. However, even within the first three values, there are different gradations: objectively speaking, a setting of DENY is more secure than ALLOW-FROM, so should there be a difference in rating? That’s a hard call, which perfectly illustrates the ambiguity in making these decisions …
When we look at the sixth security header, Content-Security-Policy (CSP), things are even more complicated. One of CSP’s primary goals was to allow you to restrict where resources are loaded from, aiming to reduce the harm of injection attacks. There are countless configurations for CSP, which include wildcards and potentially very insecure settings. To make matters even worse, several new features have been added to CSP, thereby deprecating existing security headers. Two examples are the frame-ancestors directive, which deprecates the X-Frame-Options header, and the reflected-xss directive, which deprecates the X-XSS-Protection header. So how would you evaluate the contents of a CSP policy? When is a CSP policy considered to be secure, and when is it not? What if you include unsafe-inline, but disable it after the page has loaded, like Dropbox does? The ambiguity here is off the charts. I don’t have a clear answer here, and I’m pretty sure trying to answer this question will lead to a lot of discussion among experts …
Gaming the system
These server-driven browser-enforced security policies are great to specify an application-specific security policy. However, this advantage is a direct disadvantage when evaluation the effectiveness of such a security policy. Implementing detailed evaluation mechanisms for these policies is a tremendous amount of effort, especially when you need to take all relevant cases into account.
At the moment, securityheaders.io (and the newer test version) already performs detailed checks for several of these headers. For example, you need to have a decent max-age on the Strict-Transport-Security and Public-Key-Pins headers to improve your rating. The test version also implements validation on the actual pins, to check whether they will be accepted by the browser or not.
For some of the other headers, the main decision is based on the presence of the headers and the validity of its contents. Unfortunately, this means that these checks can be exploited to game the system, resulting in a higher score can you actually deserve. Especially Content-Security-Policy is a prime candidate for gaming, since implementing a real CSP policy is less than straightforward. For example, the image below shows how a website implemented a minimal CSP policy, so that they could score better on the securityheaders.io test. By using a CSP directive to configure the browser-based XSS protection, they simply repeat the same behavior as already configured by the X-XSS-Protection header, and do not really add anything useful.
An unfortunate case of a website clearly aiming for an A without going through the effort of actually deploying CSP.
Of course, people can easily detect that you’re gaming the system, and are essentially cheating to get a better score. However, as far as the general public is concerned, you’re doing a good job security-wise, and a public shaming, like we’ve seen with the SSL Server Test, will be less likely.
I’ve been in contact with Scott about these abuses, and we had a very interesting discussion on the topic. He does acknowledge the existence of these cheats, and agrees with the fact that these people deserve to be called out on that behavior. Some of the checks on securityheaders.io could definitely benefit from more precision, which Scott is actually working on as I’m writing this article. As discussed earlier, this is not a trivial task, and in a case like CSP, it’s unlikely that there will ever be a single check that evaluates all aspects of the policy.
With all that said, I am definitely convinced that these security scanning services are priceless as far as awareness goes. Make sure you’re part of the group of people that truly values security, and use these services to keep an eye on how well you’re doing. Just pay your dues and implement the security policy in the correct way, instead of taking shortcuts.
To conclude, let me quote you Scott’s point of view from one of the mails we exchanged, with which I can only agree to the fullest:
Right now the sheer amount of education it’s doing in terms of raising awareness is really important. I’ve seen so many sites increase their scores by implementing new headers after being told of missing items, I hope it continues to have a positive impact :-)
So, if you ever run into Ivan or Scott, make sure you buy them a beer for their efforts! I know I will!
What’s your experience with these services? Do you have any insights on how to improve their accuracy, or any crazy examples on how to game the system? Let me know in the comments below!