Home > Blog > Blame it on the temps...

Blame it on the temps...

Gepubliceerd door admin on december 19, 2011

Klik hier voor de Nederlandse versie

We have only recently recovered from the 'Diginotargate' shock, yet on Dec 8th and in the days following, Webwereld published a series of articles that once more cause commotion and raise discussion about the trustworthiness of the Dutch national Public Key Infrastructure, PKIOverheid - and the PKI trust model in general.  The first article mentions the hack of a company called Gemnet, which sells Getronics digital certificates to government agencies.

Both Getronics and Gemnet are KPN subsidiaries. Getronics has been operating under the flag 'KPN Corporate Market' for a while now.  Shortly after the Gemnet website plug is pulled for investigation of the problem, a second article is published: 'KPN revokes 1000 certificates'. Despite the sensational title of the piece, the revoking  in itself does not say anything about the trustworthiness of  KPN as a Certificate Authority, however - the context raises some suspicion:

  • On Nov 4th KPN temporarily shut down the Getronics Certificate Authority because of a potential security breach. Apparently the site had been hacked 4 years ago and hacker tools had been found. It seemed however, that the Certificate Issuance process had not been compromised, and the certificate issuance is resumed a few days later.
  • The number of 1000 revoked certificates mentioned in the piece are an explosive increase compared to the on average 7 revocations per month.


In the Diginotar case a similar situation surfaced - an  increase in revocations was visible in July of this year;  the organization had desperately tried to cover up the infamous incident by certificate revocation.
A painful addition to the latter is that the Gemnet server hack had been made possible because of a very embarrassing configuration error (public phpMyAdmin access without any password set). A conspiracy has been born. The news raised enough suspicion to lead to a national political debate.

How did Webwereld discover KPN's revocation practices? The answer to this question is relatively simple: the Certification Authority publishes this information on its own website. A digital certificate has a designated lifetime, but if for any reason the certificate can no longer be trusted, the issuer then has the capability to revoke it by publishing a Certificate Revocation List (CRL). Relying parties can download this CRL from a web location in order to assess if a particular certificate is still reliable and valid. De URL can be found in the certificate's properties:

cert_cdp.png

The certificate in this example has been issued by 'Getronics CSP Organisatie CA - G2'. This Certificate Authority is part of the new PKIOverheid division, and features stronger cryptographic algorithms than its predecessor. Additionally KPN maintains a traditional PKIOverheid CA that is up for phasing out per Jan 1 2012. The latter is known as 'Getronics PinkRoccade PKIOverheid Bedrijven CA'.
Downloading the CRLs from mentioned CAs is a breeze. The easiest way to process them is by using OpenSSL. If you are interested, the command used is:

openssl crl -inform der -in LatestCRL.crl -text
 

It is clear that Webwereld's 'CA of 1000 revocations ' concerns 'Getronics CSP Organisatie CA - G2'. As mentioned earlier the high number of revocations in itself do not necessarily raise any suspicion, nor does it say anything about the integrity of the certificate authority. One can quickly come up with a few example scenarios that would justify such a number:

  • A client hast lost the privates keys related to certificates and is forced to replace them. Old certificates get revoked before new ones are issued (specifically important when certificates are used for digital signatures);
  • A company has issued smartcards to its staff, but decides to replace these before they reach their designated lifetime because the smartcards themselves cannot be considered secure anymore (for example Mifare Classic). Smartcard private keys are not exportable by design and therefore cannot be migrated onto a new card. The only solution is to create new keys and certificates. Obviously the certificates loaded on the old cards need to be revoked.

KPN provides an alternative explanation to the issue. PKIOverheid certificates are issued to two types of entities: organizations and people. Personal certificates always are issued on smartcards or USB-tokens. The certificates have been revoked as a result of KPN's so called quality assurance process. After 'Diginotargate' KPN gained many new customers. A combination of lack of experience with smartcards, the high demand and the hiring of temporary employees would have caused the low yield of valid smartcards. Since every smartcard contains 3 certificates (used for authentication, privacy and digital signatures), one invalid smartcard requires three revocations at a time. 
When analyzing the list of revocations we can clearly discern a pattern that supports this story:

Date Time Serial number
10-10-2011 13:48:41 68E76CEBFBEB52044D97CBC7AE2BA1E2
10-10-2011 13:48:44 309E93E36FB0EA46B5FE52D533299C5E
10-10-2011 13:48:46 0142C23E9E9197582AF1D7AB190605FD
10-10-2011 13:49:05 2370A8C59066C430710C12AE486B646A
10-10-2011 13:49:07 2C38CD1BE7DC6A4F101DBA9DE24E83E1
10-10-2011 13:49:10 319150241C20AAF3BFFFAC6493B6A93D
 

When certificates get revoked within a 2 to 3 second interval, we can safely assume that this is smartcard- related. Nevertheless KPN's practice of revoking smartcards during the production process seems a bit strange. Other PKIOverheid CA's don't seem to suffer from this type of production failures - their CRLs do not show similar patterns.

In my experience a smartcard production process takes the following course:

  • (Paper) application, validation and approval process for the issuance to a specific entity;
  • Card printing (Company logo, personal details, photographic image of the person etc.)
  • Production of a private/public key pair and the Certificate Signing Request containing the subject's details. This request gets sent to the Certificate Authority
  • Validation of the CSR details, comparing them to the original application form and the creation of the certificate (which is based on the CSR)
  • Loading of the certificate onto the smartcard -  This step is usually performed by an operator. The certificate issued as well as other certificates part of the trust chain are uploaded to the smartcard.

Creating a certificate basically comes down to signing a Certificate Signing Request (which involves the CA's private key). The signature means that the CA guarantees the holder of the certificates public key is genuine. It is highly remarkable that KPN as part of the production process performs the signing procedure before validating the requests details. It makes more sense to conclude the verification process with the certificate generation.

As one can see, the certificate generation step is only necessary after the final validation step. If an error is detected, no certificate will have been issued yet and no revocation step is necessary. Apparently KPN has a different philosophy.

Now that we know the ins-and-outs of KPNs smartcard production process we can focus on the remaining revoked certificates. After eliminating the personal smartcard certificates (which are identifiable due to the 3-second pattern) a significant group of revoked certificates remains, for which KPN has not yet provided an explanation: the server certificates. These types of certificates happened to be the target for the Diginotar Hacker as they can be used to intercept secured website  traffic (for example for Gmail and Twitter).

kpn_revocations_mnth.png

A similar excuse for revoking server certificates (lack of experience) would not fly, since KPN is the absolute market leader when it comes to PKIOverheid server certificates. An SSL-survey amongst Dutch government websites shows that KPN Getronics has a 91% market share.

chart_pkio_market_shares.png

Of course not all server certificates are accessible through public websites, but we can safely assume that the total number of certificates does not exceed 3000. The old Getronics PinkRoccade CA has been in place for several years, and therefore it provides good insight into normal revocation patterns. Also, during the largest part of its lifetime it was not involved in smartcard production. In the 32 months before Sep 1, 2011 only 170 server certificates were revoked. During the next 3 months 100 revocations took place. This seems a little odd for a CA that is being phased out, and which - according to Logius (the government organization that governs PKIOverheid) - is only allowed to issue new certificates by explicit permission.

gpr_revocations_mnth.png

Another interesting fact is that both CAs show a revocation-peak on Oct 10th. A logical question to ask KPN would be: what happened on that particular day?

kpn_comparison_cas.png

If we divide the revocations across periods (before and after Sep 1, 2011), more interesting facts surface. In the results a cluster of 3 or more revoked certificates within 3 seconds intervals are considered to stem from a smartcard. The isolated revocations are assumed to represent server certificates.

  Getronics PinkRoccade PKIo CA Getronics CSP Org. CA – G2
Issuance
Number of discovered server certificates 387 251
Issued before Sep 1, 2011 383 226
Issued after Sep 1, 2011 4 25
Market Share Increase after Sep 1, 2011 1,0% 11,1%
Revocation
CRL timespan (months) 35 17
Number of revocations (Smartcards) 43 291
Revoked Smartcards before Sep 1, 2011 40 16
Revoked Smartcards after Sep 1, 2011 3 275
Avg rev/mnth before Sep 1, 2011 1,3 1,1
Avg rev/mnth after Sep 1, 2011 1,0 91,7
Number of revocations (Server Certs) 270 305
Revoked Server Certs before Sep 1, 2011 170 67
Revoked Server Certs after Sep1, 2011 100 238
Avg rev/mnth before Sep 1, 2011 5,3 4,8
Avg rev/mnth after Sep 1, 2011 33,3 79,3
% increase rev/mnth 627% 1658%

The market share increase for server certificates is 11% at best. One would expect that this would cause a roughly similar increase in revocations. However, the table shows an explosive growth factor of up to 16 times the usual number of revocations per month.

The preliminary conclusion to the above is that KPN's explanation of its revocation procedure raises more questions than it answers. My advice to CA's would be to always be honest and open about this procedure; especially because within the certificate business it is all about trust. In KPN's case any doubt in the CA's integrity can easily be taken away by proving that the revoked certificates were issued to Dutch government organizations. However, this would prove KPN's poor certificate process,  yet it would immediately remove any suspicion of a hack or of the issuance of rogue certificates. Even if or when a hack would have occurred, it is better to be open, honest and transparent about it. Comodo is a great example of a CA that was compromised, but that managed to limit damages by communicating openly.

If at some point a certificate issued to Google, Microsoft, Yahoo or alike - signed by Getronics/KPN - would turn up, this would truly be a full-blown 'Diginotargate 2'. It would mean the destruction of PKIOverheid and it would severely damage the reputation of the Netherlands abroad.

Comments: