I've been struggling for the last week trying to get some cryptography code to behave the same on Windows and under Java. Yes, I searched Google, and I was even let down by my new favorite Stack Overflow. With all those misses, I thought this would be the perfect opportunity to improve my page ranking by trying to provide legitimate content that was useful to the general programming public.

The problem was trying to generate digital signatures in non-managed C++ under Windows and in Java and being able to verify the signature on the remote side, possibly in the other environment.

On the Java side we were using the Bouncy Castle libraries through the SPI layer. It seemed to be working perfectly and we were able to sign and verify between Java clients and servers. If we couldn't, that would mean we were doing something horribly wrong and the whole affair would be in jeopardy.

With the Java side working nicely, we moved on to getting the C++ and C# code working. For C#, we used the .NET RSACryptoServiceProvider. We generate the SHA-1 hash and then pass it to the SignHash method to generate the encrypted signature.

Now, on to C++. Our initial attempt used the Windows CryptoAPI, but I think it is at too low a layer and when we decrypted the signature, the byte count was completely wrong. The Java signatures came out to be 128 bytes for a 1024 bit key, but the WinCrypt came out to 127 bytes.

Not seeing a promising road ahead, we decided to write a wrapper DLL that would allow the unmanaged C++ code to call into the .NET RSACryptoServiceProvider to use the SignHash method we were using from the C# code. Now when we would receive the signature, the decrypted byte count was 35 which was closer to the 20 we thought we needed to match the SHA-1 digest, but we had no idea where the extra 15 bytes originated.

In the Java code, we were generating the signatures in a manual fashion, thinking that would be closer to the "metal" and allow easier reproduction on the C++ side. We generated the SHA-1 digest and then encrypted it with the private key. When we decrypted the .NET generated signature, the 35 bytes contained the same SHA-1 hash, but the 15 extra bytes were ahead of it. Sensing that we were really close, I started to look through the Java crypto API to see if anything else might be more suitable.

In the release notes for Bouncy Castle, I saw a reference to signature algorithms and SHA1WithRSA was listed as an option. I rewrote the signing code using the signature algorithm and low and behold, the signatures started matching. After spending a week trying to debug this we finally found our solution and everything is working great now.

But, what about those 15 bytes? Well dear readers let me point you to the special sauce. RFC 3447 describes how RSA cryptography is supposed to be done—properly. If you're in a rush, trying to meet a deadline it's not the sort of document you want to read. Heavy on math and a little difficult to follow, it's not the sort of thing you turn to first. I found a link to the RFC from another search and decided now that we had it working I would see if I could figure out where the 15 bytes came from. Scanning through the RFC, I came across the signature section. It turns out, to generate the signature, after generating the digest, you prepend an OID for the digest algorithm to the digest and then the entire thing is encrypted with the RSA private key. In the case of SHA-1 the OID is 15 bytes.

This was one of those great triumphs that I love about programming. You're stuck on a problem working to solve it and when you do, you've learned something more about the larger world of programming. Something probably known by many others, but you've discovered it anew and now it's yours to keep.