The trailing space matters

linux devops

TL;DR: In the last post, I was bit by the missing dot, and this time, an extra trailing space.

We recently secured some of our internal web services with HTTPS to meet the PCI compliance, and the SSL verification is also enforced in the client side. Since the certificate is self-signed, we need to override REQUSTS_CA_BUNDLE environment variable for the awesome requests library to use the specified certificate to verify the peer.

In my humble opinion, /etc/environment is a more sane place to setup the global environment variable as it is shell-agnostic. The only drawback is /etc/environment is evaluated when PAM takes effect; to make the change take effect, you have to exit all the processes owned by the user, logout the session, and login again.

We did that, and it works perfectly fine in our staging environment, but it did not work in the testing environment even for the same configure file. In the strace log, it shows that the python process failed to open the certificate:

open("/path/to/myca.crt\" ", O_RDONLY) = -1 ENOENT (No such file or directory)

And this can be reproduced by the python REPL:

>>> import os
>>> print "..%s.." % os.environ.get('REQUESTS_CA_BUNDLE')
../path/to/myca.crt" ..

and also printenv:

printenv REQUESTS_CA_BUNDLE
/path/to/myca.crt

cat /etc/environment
REQUESTS_CA_BUNDLE="/path/to/myca.crt"

I then dumped the /etc/environment:

$ hexdump -C /etc/environment
00000000  52 45 51 55 45 53 54 53  5f 43 41 5f 42 55 4e 44  |REQUESTS_CA_BUND|
00000010  4c 45 3d 22 2f 70 61 74  68 2f 74 6f 2f 6d 79 63  |LE="/path/to/myc|
00000020  61 2e 63 72 74 22 20 0a                           |a.crt" .|
00000028

There exists a trailing space after the double quote, does it make a difference?

Oh, yeah. Once the trailing space was removed, the next time I logged in, everything just worked.

It is still unclear why the old config file worked in the staging environment. The only plausible explanation is that the testing environment runs a more recent version of Ubuntu server, which interprets /etc/environment more literally.