Press "Enter" to skip to content

Tag: ssl

cPanel DNSOnly Server Hostname SSL Renewal Issues – Fix

This has turned into a little “story” about how I resolved an issue with Sectigo SSL certificates failing HTTP DCV validation on a cPanel DNS Only server. It’s a bit long, but you can just jump to the conclusion at the end if you want to – or go to individual sections.

Read more

System Setup

  • I have a cPanel DNSOnly server setup running secondary DNS (gee, would have guessed by the product name? ? ) for a server. It’s setup to be a “read-only” server and not to send changes (synchronisation) to the main server.
  • Because that server should only offer DNS facilities to the “outside world”/general internet, it’s been locked down for security. Only a few select IP addresses can access SSH (on port 22) or WHM (on port 2087) – the entire internet can access UDP/TCP ports 53 (DNS) – but apart from that, there is no access whatsoever to the server.
  • The hostname of the server is in a domain name hosted by an external provider
  • The server has HSTS (HTTP Strict Transport Security) enabled which means – if the certificate has a fault – WHM will not be accessible. I’ll therefore do everything via the command line “just in case”.
  • For the explanation, the host name of the server will be ns3.example.com and the server IP will be 198.51.100.23

What happened?

After about 65 days of the server being setup in the above configuration, I started receiving the following message (why 65 days? the SSL certificates are normally issued for 90 days and had cPanel’s contact notifications for “cPanel service SSL certificate warnings - This option indicates that a warning was generated while checking the cPanel service SSL certificates.” enabled (cPanel starts auto renewing when the certificate has 25 days or less left).

Subject: [ns3.example.com ] ? 1 service generated warnings while checking SSL certificates

The following cPanel service generated warnings from the checkallsslcerts script.
? cpanel
The system cannot install the fetched certificate (EXPIRES_SOON).
The system failed to acquire a signed certificate from the cPanel Store because of the following error: All HTTP and DNS DCV preflight checks failed!
This notice is the result of a request from “/usr/local/cpanel/bin/checkallsslcerts”.
The system generated this notice on xxxx.
“cPanel service SSL certificate warnings” notifications are currently configured to have an importance of “Medium”. You can change the importance or disable this type of notification in WHM’s Contact Manager at: https://ns3.example.com:2087/scripts2/editcontact?event=SSL::CheckAllCertsWarnings
Do not reply to this automated message.

Why did this happen?

The cPanel Inc provided SSL certificate (provided via Sectigo) was due to expire within 30 days, but the server was unable to renew it because the security checks (Domain Control Validation – DCV) could not verify the server was in control of the hostname ns3.example.com

It did try confirming it over DNS, but as the DNS was remotely/externally hosted, it could not make the necessary changes “in real time”. It then also tried confirming it over HTTP, but this failed as – being a DNS server – this server does not run a web server such as Apache or nginx.

First checkallsslcerts test: What is failing?

The first step was to try and see what was happening. This was done by running the cPanel checkallsslcerts command and seeing the output (some bits shortened):

# /usr/local/cpanel/bin/checkallsslcerts --allow-retry --verbose
The system will check for the certificate for the “cpanel” service.
...
The system will attempt to install a certificate for the “cpanel” service from the cPanel store.
Setting up HTTP DCV (/usr/local/apache/htdocs/.well-known/pki-validation/xxxxxxxx.txt) …
… complete.
Setting up DNS DCV for “ns3.example.com” …
… complete.
Attempting DNS DCV preflight checks …
ns3.example.com: DNS DCV preflight check failed; falling back to HTTP …
....
ns3.example.com: Attempting HTTP DCV preflight check …
The system queried for a temporary file at “http://ns3.example.com/.well-known/pki-validation/xxxxxxxx.txt”, but the web server responded with the following error: 401 (Access Denied). A DNS (Domain Name System) or web server misconfiguration may exist.
Undoing HTTP DCV setup …
… complete.
Undoing DNS DCV setup …
… complete.
[WARN] The system failed to acquire a signed certificate from the cPanel Store because of the following error: All HTTP and DNS DCV preflight checks failed!

As we can see, the failure happened within the “preflight” section (where the server checks itself that things are working before it makes a call out). Let’s see if we can work out why – we’ll ignore the DNS DCV method as the server can’t control the external domain name.

Checking the web path

The file cPanel creates for DCV preflight checks (and also for the actual certificate request) is removed at the end of the run (“Undoing HTTP DCV setup...“), so we’ll have to make a temporary file to test – the checkallsslcerts system used the folder /usr/local/apache/htdocs/.well-known/pki-validation , so let’s make a temporary file in there:

# touch /usr/local/apache/htdocs/.well-known/pki-validation/my-test-file.txt

and try accessing it via a web browser at http://ns3.example.com/.well-known/pki-validation/my-test-file.txt . Oh. it worked. Odd… Well, not really – my office IP address is “allowed” to access nearly everything (the “old terminology” would be “whitelisted”).

Checking the web path from the server

But the server is complaining about the preflight – so let’s try the request from the server itself.

# curl -I http://ns3.example.com/.well-known/pki-validation/test.txt
HTTP/1.0 401 Access Denied
Connection: close
Content-Type: text/html; charset="utf-8"
Date: xxxx
Cache-Control: no-cache, no-store, must-revalidate, private
Pragma: no-cache
X-Error-Message: Access Denied
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
Content-Length: 5121

So the URL is blocked from being accessed by the server. But by what? Because we know that the path is /usr/local/apache/htdocs/ (and not /var/www/html/ or /home/xxxx/public_html ), and the fact there is no “Server:” identifier included in the returned headers, there stands a high chance of this being served from cPanel itself with it’s built in “mini webserver” (which is normally used to serve the WHM, cPanel and Webmail UIs).

We can check this by using cPanel’s access log:

# grep 401 /usr/local/cpanel/logs/access_log
198.51.100.23- - [xxxxxx -0000] "-" 401 0 "-" "-" "-" "-" 80

So, yes, it is cPanel blocking it. We can confirm this by using cPanel’s error log:

# grep 198.51.100.23 /usr/local/cpanel/logs/error_log
Dropping connection from 198.51.100.23 because of host access control at cpsrvd.pl line 4128.

Brilliant – and we’ve got a clue of what we need to allow – host access control.

Host Access Control

As you may have seen from the “Host Access Control” web pages in WHM, you are required to enter a “TCP Wrapper service name” (or “ALL“), an IP address (or “ALL“) and whether you are wanting to “allow” or “deny” access. This file is stored as /etc/hosts.allow .

But the problem we have is we don’t know which service name to enter. I tried “ALL” and that worked, but that would allow access to everything – which I didn’t want. I then tried all the “known service names”:

  • snmp (“SNMP Service”)
  • sshd (“SSH Service”)
  • pop3 (“Pop3 Service Daemon”)
  • domain (“DNS Services”)
  • auth (“Ident Service”)
  • cpaneld (“cPanel Service Daemon”)
  • postgresql (“PostgreSQL Service”)
  • smtp (“SMTP Service Daemon”)
  • whostmgrd (“Web Host Manager Service Daemon”)
  • cpdavd (“WebDav/WebDisk Service Daemon”)
  • telnet (“Telnet Service”)
  • ftp (“Ftp Server”)
  • mysql (“MySQL Server”)
  • imap (“Imap Service Daemon”)
  • webmaild (“WebMail Service Daemon”)

(taken from a WHM “Host Access Control” page I could access and examining the source),

but none of them worked. So which service is it?

Host Access Control – Finding The Service

In /etc/hosts.allow I added the following line near the bottom of the file before the “ALL : ALL : deny” line:

ALL : 198.51.100.23 : spawn /bin/echo `date` %c %d >> /root/test.txt

This causes the TCP wrapper service to launch a process (spawn) echoing the current date, the IP address (%c) and the service name (%d) to the file /root/test.txt [ see the Softpanorama page for TCP Wrappers under “Shell Commands”). I then re-tried the curl command from the server and looked at /root/test.txt

# cat /root/test.txt
xxxxx UTC 2022 198.51.100.23 cphttpd

Bingo – it’s the “cphttpd” service which isn’t listed in WHM: no wonder that “ALL” would allow it but none of the other services did. Now let’s remodify /etc/hosts.allow to allow that service for our 198.51.100.23 IP address:

cphttpd: 198.51.100.23 : allow

Repeat the curl test a third (or forth?) time and bingo!

# curl -I http://ns3.example.com/.well-known/pki-validation/test.txt
HTTP/1.1 200 OK

Success – we’ve got the server accessing itself!

We can now delete the test file:

# rm /usr/local/apache/htdocs/.well-known/pki-validation/my-test-file.txt

Second checkallsslcerts test – Checking preflight

Let’s try checkallsslcerts again and see how far we get:

# /usr/local/cpanel/bin/checkallsslcerts --allow-retry --verbose
The system will check for the certificate for the “cpanel” service.
...
The system will attempt to install a certificate for the “cpanel” service from the cPanel store.
Setting up HTTP DCV (/usr/local/apache/htdocs/.well-known/pki-validation/xxxxxxxx.txt) …
… complete.
Setting up DNS DCV for “ns3.example.com” …
… complete.
Attempting DNS DCV preflight checks …
....
ns3.example.com: Attempting HTTP DCV preflight check …
        … success!
...
Succeeded domains: 1
Failed domains: 7
Undoing HTTP DCV setup …
        … complete.
Undoing DNS DCV setup …
        … complete.
Setting up HTTP DCV (/usr/local/apache/htdocs/.well-known/pki-validation/xyzxyz.txt) …
        … complete.
Setting up DNS DCV for “ns3.example.com” …
        … complete.

Requesting certificate from cPStore …
        Order submitted. (Order item ID: 12345)
... [short wait] ...
The cPanel Store is processing the hostname certificate request.
The system will check the cPanel Store again in an hour to see if the cPanel Store issued the certificate.

Brilliant – we passed the preflight check with “success!”, had 1 succeeded domain (don’t worry about the “failed domains: 7” – that’s things like webmail.ns3.example.com cpanel.ns3.example.com etc etc), and we can see it’s setup the proper HTTP DCV validation with a different filename ( xyzxyz.txt ) which does actually correspond to the md5 hash of the certificate request. And we see it’s been submitted to the cPStore, but there is a processing delay.

Server can access it, but what about the certificate issuer?

Ah yes – so far, we’ve just fixed the preflight system allowing the server to check itself – but not allowed any third parties/services access. So which IP addresses do we need to allow/whitelist? Well, as soon as the “Order submitted” came up, cPanel’s cPStore passed the request to Sectigo to process and they made a request back. Let’s have a look in the access log to see if it was blocked.

# grep 401 /usr/local/cpanel/logs/access_log
91.199.212.132 - - [xxxxx] "-" 401 0 "-" "-" "-" "-" 80

Oh – an IP address we don’t recognise as one of ours. Now, it could be Sectigo or it could just be someone else randomly testing if we have a website. Let’s do an lookup to check.

Whois Lookups with Team Cymru

So how can we verify that we want to trust that IP address? Whilst we could just use rDNS (reverse DNS) to check:

# host 91.199.212.132
132.212.199.91.in-addr.arpa domain name pointer secure.trust-provider.net.

that doesn’t really help much (who are trust-provider.net? There’s no website at that URL and rDNS is reasonably easy to fake).

Instead, we’ll use WHOIS with the Team Cyrmu WHOIS service to lookup the “Autonomous System Number” (ASN) details which will be a lot more authoritative (as it’ll list who “owns” that entire netblock and the size of it). First we’ll need to install (using yum), the whois command but then we are away!

# yum install whois
# whois -h whois.cymru.com "-v 91.199.212.132"
Warning: RIPE flags used with a traditional server.
AS      | IP               | BGP Prefix          | CC | Registry | Allocated  | AS Name
48447   | 91.199.212.132   | 91.199.212.0/24     | GB | ripencc  | 2008-02-22 | SECTIGO, GB

We can see that the IP address 91.199.212.132 – according to the RIPE NCC Registry – belongs to AS48447 who are “Sectigo, GB”. And as we know cPanel Inc uses Sectigo – we’ve got the right company! (btw Sectigo used to be known as Comodo CA).

Finding Sectigo’s netblocks using PWhois

Whilst we could just allow the listed prefix/netblock of 91.199.212.0/24 access to the “cphttpd” service – there’s a good chance Sectigo have other netblocks allocated to them/their ASN. There’s another service we could utilise to find all their netblocks – PWhois.org (yes, we could have used PWhois for the initial IP to ASN lookup as well, but I didn’t so there 😉 ).

# whois -h whois.pwhois.org "routeview source-as=48447"
Origin-AS: 48447
    Prefix            | Create-Date              | Modify-Date              | Originated-Date          | Next-Hop        | AS-Path
*>      5.183.44.0/22 |     Jul 18 2022 00:00:03 |     Jul 18 2022 00:00:03 |     Jul 09 2022 05:54:02 |  208.115.137.35 | 8220 1299 174 48447
*>    91.199.212.0/24 |     Jul 18 2022 00:00:03 |     Jul 18 2022 00:00:03 |     Jul 09 2022 05:54:02 |  208.115.137.35 | 8220 1299 174 48447
*>    91.209.196.0/24 |     Jul 18 2022 00:00:03 |     Jul 18 2022 00:00:03 |     Jul 09 2022 05:54:02 |  208.115.137.35 | 8220 1299 174 48447
*>     91.212.12.0/24 |     Jul 18 2022 00:00:03 |     Jul 18 2022 00:00:03 |     Jul 09 2022 05:54:02 |  208.115.137.35 | 8220 1299 174 48447

Now we have all the netblocks allocated to AS48447 (at least all of those found by pwhois and it’s possible Sectigo have other ASN allocations with RIPE and even with other registries, but we’ll just ignore those facts for now 😉 ).

Allowing those netblocks

Now just edit /etc/hosts.allow again to add those ranges at the bottom (before the “ALL : ALL : deny” entry) allowing access to cphttpd (remembering to keep our server IP address (198.51.100.23) listed as otherwise the preflights will fail again): We’ll also add some comments ( starting with # ) just in case we forget why we added them.

cphttpd : 198.51.100.23 : allow
# previous IP is our server IP address
cphttpd : 5.183.44.0/22 : allow
# previous address is Sectigo for web DCV access
cphttpd : 91.199.212.0/24 : allow
# previous address is Sectigo for web DCV access
cphttpd : 91.209.196.0/24 : allow
# previous address is Sectigo for web DCV access
cphttpd : 91.212.12.0/24 : allow
# previous address is Sectigo for web DCV access
ALL : ALL : deny

Now just wait for the certificate to be issued (if, after an hour, it hasn’t been – then just rerun checkallsslcerts )

You can see the actual check from Sectigo if you monitor /usr/local/cpanel/logs/access_log :

# tail /usr/local/cpanel/logs/access_log
...
91.199.212.132 - - [xxxxx -0000] "GET /.well-known/pki-validation/xyzxyz.txt HTTP/1.1" 200 0 "-" "Sectigo DCV" "-" "-" 80

Allow the list cPanel Inc provide

For some reason, I didn’t see this handy page “What IP addresses do Sectigo DCV requests originate from?“on the cPanel knowledgebase before doing all this. So just change your /etc/hosts.allow file to have – at the bottom –

cphttpd : 198.51.100.23 : allow
# previous IP is our server IP address
cphttpd : 178.255.81.12 : allow
# previous address is Sectigo for web DCV access
cphttpd : 178.255.81.13 : allow
# previous address is Sectigo for web DCV access
cphttpd : 91.199.212.132 : allow
# previous address is Sectigo for web DCV access
cphttpd : 199.66.201.132 : allow
# previous address is Sectigo for web DCV access
cphttpd : 91.199.212.52 : allow
# previous address is Sectigo for web DCV access
cphttpd: [2a02:1788:400:1ce4::/64] : allow
# previous address is Sectigo for web DCV access
ALL : ALL : deny

Note the formatting of the IPv6 address.

All done!

[Techy] Getting cURL to work with Let’s Encrypt: unable to get local issuer certificate error

On my test Debian jessie 8.2 and a staging server Ubuntu trusty 14.04, I had problems being able to use cURL to fetch data from a remote HTTPs site which was secured using a free Let’s Encrypt certificate (this problem manifested itself via both PHP 7 cURL functions and curl directly).

An example of the error is:

curl --verbose https://helloworld.letsencrypt.org/
...
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: http://curl.haxx.se/docs/sslcerts.html

To fix this, I found the following steps worked:
sudo apt-get install --reinstall ca-certificates
to ensure you have the latest certificates by your distribution – this might help you, but it didn’t assist me 🙁

So I downloaded the Let’s Encrypt root certificates and forced a rebuild of the certificate store:

sudo curl https://letsencrypt.org/certs/isrgrootx1.pem.txt -o /usr/local/share/ca-certificates/isrgrootx1.crt
sudo curl https://letsencrypt.org/certs/letsencryptauthorityx1.pem.txt -o /usr/local/share/ca-certificates/letsencryptauthorityx1.crt
sudo curl https://letsencrypt.org/certs/letsencryptauthorityx2.pem.txt -o /usr/local/share/ca-certificates/letsencryptauthorityx2.crt
sudo curl https://letsencrypt.org/certs/lets-encrypt-x1-cross-signed.pem.txt -o /usr/local/share/ca-certificates/letsencryptx1.crt
sudo curl https://letsencrypt.org/certs/lets-encrypt-x2-cross-signed.pem.txt -o /usr/local/share/ca-certificates/letsencryptx2.crt
sudo curl https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem.txt -o /usr/local/share/ca-certificates/letsencryptx3.crt
sudo curl https://letsencrypt.org/certs/lets-encrypt-x4-cross-signed.pem.txt -o /usr/local/share/ca-certificates/letsencryptx4.crt
sudo dpkg-reconfigure ca-certificates

and everything worked!

If you are running Java, you may need to also add the certificates to the Java Keytool:
keytool -trustcacerts -keystore $JAVA_HOME/jre/lib/security/cacerts -storepass changeit -noprompt -importcert -file /usr/local/share/ca-certificates/isrgrootx1.crt
keytool -trustcacerts -keystore $JAVA_HOME/jre/lib/security/cacerts -storepass changeit -noprompt -importcert -file /usr/local/share/ca-certificates/letsencryptauthorityx1.crt

(change the storepass password [default of “changeme”] to whatever is relevant to you)

Techy: How I investigated Counterpath Bria’s licensing issue

On Saturday the 12th of February 2010, myself and many other users of the popular Counterpath Bria VOIP Software started getting the message “The data received from the license server appears to be corrupted. Please try again and if the problem persists, contact Counterpath Support.”, but without any idea of what caused it. Many people contacted Counterpath’s support forums, but as of the time of posting the only official status we have is “It is being investigated”. However, I took it upon myself to try and figure out what had gone wrong and then if I could “work around” the issue.

I knew from the error message that Bria was trying to contact the licensing server to validate the license, so the first step was to investigate what was being transmitted. Luckily, I had the very handy Charles Proxy and after installing its CA Certificate (so Bria didn’t complain about an invalid certificate authority when retrieving the data via Charles), I could see the communicaitons.

Bria made a post to https://secure.counterpath.com/Service/LicensePool.sv/RegisterClientBinding with my license key, a hash of something on my machine, the type of product I’m checking the license for (Bria), the time my computer thinks it is, and identifying hashes of my machine (in particular, hard disk, motherboard and Mac address) – this data is sent to try and reduce “license sharing”. In return, Counterpath sent back the same information, including an expiration date and duration of the license (30 days), instructions to recheck the license after 24 hours and then every 4 hours (+/- 30%), confirmation the license if valid and then they signed the response using an X509 digital signature. Why have they signed it? Well, otherwise it would be reasonably simple for somebody of about my technical knowledge to be able to intercept the SSL stream and modify the data to always have approved licences (and, no, I’m not going to say how I would do this – but I can think of 2 ways straight way).

As all the other data looked correct (Counterpath used descriptive XML tags which did really help in this process: although since it’s just communicating between their servers and their products, I would have personally obfuscation it just for a little more ‘security’: yes, I know “security by obfuscation” isn’t good security practice, but sometimes “every little helps”), I then investigated the X509 certificate. I copied and pasted it into a new text document, called it “x509.crt” and just let Windows’ certificate explorer show be the details – and it was immediately obvious what the fault was. The expiration date on the certificate was dated 2 days ago.

So about 5-10 minutes of investigation and I knew to roll back my computer clock to pre 12th February 2010. Run Bria to get it to reacquire a new license and viola everything working (well, hopefully for 24 hours until the license starts revalidating). I just wish Counterpath had actually provided the “roll back your clock” workaround on their forum: Bria is an excellent product and I’m just disappointed that because somebody dropped the ball and forgot to renew a 2 year old X509 certificate (which would be easily done as that length of time people would have left and the details would have just been forgotten to be passed on), their reputation and good product range may be tainted. I’m guilty myself of letting the SSL certificate on my online dance wear site expire for a couple of days (mainly because the certificate provider wouldn’t let me renew the certificate until after it had expired!) so I know how easily it can be done – I now have a 3 year certificate so it might again happen to me in 3 years time.

What has this taught us: If you use a secure certificate (for your website or code), keep a note of the date it is due to expire and set reminders. Your certificate provider might send you emails to remind you – but don’t rely on them. Communicate with your customers and if a problem could take more than a couple of hours to fix, propose a workaround (such as rolling back your computer’s clock) – they might not like it, but at least you are doing “something”. And signing license files with your own digital signature is a good idea (yep, I bet you weren’t expecting that!) as it stops people easily bypassing it – but just remember to keep your digital signature up to date!

[n.b. might just be worth mentioning that I investigated this all on my own – Counterpath or any third party did not ask me to investigate it. However, within an hour of me posting on their forums with the information I had found, the problem was fixed. Coincidence or were they already waiting for the new certificate to be approved/installed? I don’t know, but I like to speculate 😉 )