This has turned into a little “story” about how I resolved an issue with Sectigo SSL certificates failing HTTP DCV validation on a cPanel DNS Only server. It’s a bit long, but you can just jump to the conclusion at the end if you want to – or go to individual sections.
Read more- What happened?
- Why did this happen?
- First checkallsslcerts test: What is failing?
- Checking the web path
- Checking the web path from the server
- Host Access Control
- Host Access Control – Finding The Service
- Second checkallsslcerts test – Checking preflight
- Server can access it, but what about the certificate issuer?
- Whois Lookups with Team Cymru
- Finding Sectigo’s netblocks using PWhois
- Allowing those netblocks
- Allow the list cPanel Inc provide
System Setup
- I have a cPanel DNSOnly server setup running secondary DNS (gee, would have guessed by the product name? ? ) for a server. It’s setup to be a “read-only” server and not to send changes (synchronisation) to the main server.
- Because that server should only offer DNS facilities to the “outside world”/general internet, it’s been locked down for security. Only a few select IP addresses can access SSH (on port 22) or WHM (on port 2087) – the entire internet can access UDP/TCP ports 53 (DNS) – but apart from that, there is no access whatsoever to the server.
- The hostname of the server is in a domain name hosted by an external provider
- The server has HSTS (HTTP Strict Transport Security) enabled which means – if the certificate has a fault – WHM will not be accessible. I’ll therefore do everything via the command line “just in case”.
- For the explanation, the host name of the server will be ns3.example.com and the server IP will be 198.51.100.23
What happened?
After about 65 days of the server being setup in the above configuration, I started receiving the following message (why 65 days? the SSL certificates are normally issued for 90 days and had cPanel’s contact notifications for “cPanel service SSL certificate warnings - This option indicates that a warning was generated while checking the cPanel service SSL certificates.” enabled (cPanel starts auto renewing when the certificate has 25 days or less left).
Subject: [ns3.example.com ] ? 1 service generated warnings while checking SSL certificates
The following cPanel service generated warnings from the checkallsslcerts script.
? cpanel
The system cannot install the fetched certificate (EXPIRES_SOON).
The system failed to acquire a signed certificate from the cPanel Store because of the following error: All HTTP and DNS DCV preflight checks failed!
This notice is the result of a request from “/usr/local/cpanel/bin/checkallsslcerts”. The system generated this notice on xxxx.
“cPanel service SSL certificate warnings” notifications are currently configured to have an importance of “Medium”. You can change the importance or disable this type of notification in WHM’s Contact Manager at: https://ns3.example.com:2087/scripts2/editcontact?event=SSL::CheckAllCertsWarnings
Do not reply to this automated message.
Why did this happen?
The cPanel Inc provided SSL certificate (provided via Sectigo) was due to expire within 30 days, but the server was unable to renew it because the security checks (Domain Control Validation – DCV) could not verify the server was in control of the hostname ns3.example.com
It did try confirming it over DNS, but as the DNS was remotely/externally hosted, it could not make the necessary changes “in real time”. It then also tried confirming it over HTTP, but this failed as – being a DNS server – this server does not run a web server such as Apache or nginx.
First checkallsslcerts test: What is failing?
The first step was to try and see what was happening. This was done by running the cPanel checkallsslcerts command and seeing the output (some bits shortened):
# /usr/local/cpanel/bin/checkallsslcerts --allow-retry --verbose The system will check for the certificate for the “cpanel” service. ... The system will attempt to install a certificate for the “cpanel” service from the cPanel store. Setting up HTTP DCV (/usr/local/apache/htdocs/.well-known/pki-validation/xxxxxxxx.txt) … … complete. Setting up DNS DCV for “ns3.example.com” … … complete. Attempting DNS DCV preflight checks … ns3.example.com: DNS DCV preflight check failed; falling back to HTTP … .... ns3.example.com: Attempting HTTP DCV preflight check … The system queried for a temporary file at “http://ns3.example.com/.well-known/pki-validation/xxxxxxxx.txt”, but the web server responded with the following error: 401 (Access Denied). A DNS (Domain Name System) or web server misconfiguration may exist. Undoing HTTP DCV setup … … complete. Undoing DNS DCV setup … … complete. [WARN] The system failed to acquire a signed certificate from the cPanel Store because of the following error: All HTTP and DNS DCV preflight checks failed!
As we can see, the failure happened within the “preflight” section (where the server checks itself that things are working before it makes a call out). Let’s see if we can work out why – we’ll ignore the DNS DCV method as the server can’t control the external domain name.
Checking the web path
The file cPanel creates for DCV preflight checks (and also for the actual certificate request) is removed at the end of the run (“Undoing HTTP DCV setup...“), so we’ll have to make a temporary file to test – the checkallsslcerts system used the folder /usr/local/apache/htdocs/.well-known/pki-validation , so let’s make a temporary file in there:
# touch /usr/local/apache/htdocs/.well-known/pki-validation/my-test-file.txt
and try accessing it via a web browser at http://ns3.example.com/.well-known/pki-validation/my-test-file.txt . Oh. it worked. Odd… Well, not really – my office IP address is “allowed” to access nearly everything (the “old terminology” would be “whitelisted”).
Checking the web path from the server
But the server is complaining about the preflight – so let’s try the request from the server itself.
# curl -I http://ns3.example.com/.well-known/pki-validation/test.txt HTTP/1.0 401 Access Denied Connection: close Content-Type: text/html; charset="utf-8" Date: xxxx Cache-Control: no-cache, no-store, must-revalidate, private Pragma: no-cache X-Error-Message: Access Denied X-Frame-Options: SAMEORIGIN X-Content-Type-Options: nosniff Content-Length: 5121
So the URL is blocked from being accessed by the server. But by what? Because we know that the path is /usr/local/apache/htdocs/ (and not /var/www/html/ or /home/xxxx/public_html ), and the fact there is no “Server:” identifier included in the returned headers, there stands a high chance of this being served from cPanel itself with it’s built in “mini webserver” (which is normally used to serve the WHM, cPanel and Webmail UIs).
We can check this by using cPanel’s access log:
# grep 401 /usr/local/cpanel/logs/access_log 198.51.100.23- - [xxxxxx -0000] "-" 401 0 "-" "-" "-" "-" 80
So, yes, it is cPanel blocking it. We can confirm this by using cPanel’s error log:
# grep 198.51.100.23 /usr/local/cpanel/logs/error_log Dropping connection from 198.51.100.23 because of host access control at cpsrvd.pl line 4128.
Brilliant – and we’ve got a clue of what we need to allow – host access control.
Host Access Control
As you may have seen from the “Host Access Control” web pages in WHM, you are required to enter a “TCP Wrapper service name” (or “ALL“), an IP address (or “ALL“) and whether you are wanting to “allow” or “deny” access. This file is stored as /etc/hosts.allow .
But the problem we have is we don’t know which service name to enter. I tried “ALL” and that worked, but that would allow access to everything – which I didn’t want. I then tried all the “known service names”:
- snmp (“SNMP Service”)
- sshd (“SSH Service”)
- pop3 (“Pop3 Service Daemon”)
- domain (“DNS Services”)
- auth (“Ident Service”)
- cpaneld (“cPanel Service Daemon”)
- postgresql (“PostgreSQL Service”)
- smtp (“SMTP Service Daemon”)
- whostmgrd (“Web Host Manager Service Daemon”)
- cpdavd (“WebDav/WebDisk Service Daemon”)
- telnet (“Telnet Service”)
- ftp (“Ftp Server”)
- mysql (“MySQL Server”)
- imap (“Imap Service Daemon”)
- webmaild (“WebMail Service Daemon”)
(taken from a WHM “Host Access Control” page I could access and examining the source),
but none of them worked. So which service is it?
Host Access Control – Finding The Service
In /etc/hosts.allow I added the following line near the bottom of the file before the “ALL : ALL : deny” line:
ALL : 198.51.100.23 : spawn /bin/echo `date` %c %d >> /root/test.txt
This causes the TCP wrapper service to launch a process (spawn) echoing the current date, the IP address (%c) and the service name (%d) to the file /root/test.txt [ see the Softpanorama page for TCP Wrappers under “Shell Commands”). I then re-tried the curl command from the server and looked at /root/test.txt
# cat /root/test.txt xxxxx UTC 2022 198.51.100.23 cphttpd
Bingo – it’s the “cphttpd” service which isn’t listed in WHM: no wonder that “ALL” would allow it but none of the other services did. Now let’s remodify /etc/hosts.allow to allow that service for our 198.51.100.23 IP address:
cphttpd: 198.51.100.23 : allow
Repeat the curl test a third (or forth?) time and bingo!
# curl -I http://ns3.example.com/.well-known/pki-validation/test.txt HTTP/1.1 200 OK
Success – we’ve got the server accessing itself!
We can now delete the test file:
# rm /usr/local/apache/htdocs/.well-known/pki-validation/my-test-file.txt
Second checkallsslcerts test – Checking preflight
Let’s try checkallsslcerts again and see how far we get:
# /usr/local/cpanel/bin/checkallsslcerts --allow-retry --verbose
The system will check for the certificate for the “cpanel” service.
...
The system will attempt to install a certificate for the “cpanel” service from the cPanel store.
Setting up HTTP DCV (/usr/local/apache/htdocs/.well-known/pki-validation/xxxxxxxx.txt) …
… complete.
Setting up DNS DCV for “ns3.example.com” …
… complete.
Attempting DNS DCV preflight checks …
....
ns3.example.com: Attempting HTTP DCV preflight check …
… success!
...
Succeeded domains: 1
Failed domains: 7
Undoing HTTP DCV setup …
… complete.
Undoing DNS DCV setup …
… complete.
Setting up HTTP DCV (/usr/local/apache/htdocs/.well-known/pki-validation/xyzxyz.txt) …
… complete.
Setting up DNS DCV for “ns3.example.com” …
… complete.
Requesting certificate from cPStore …
Order submitted. (Order item ID: 12345)
... [short wait] ...
The cPanel Store is processing the hostname certificate request.
The system will check the cPanel Store again in an hour to see if the cPanel Store issued the certificate.
Brilliant – we passed the preflight check with “success!”, had 1 succeeded domain (don’t worry about the “failed domains: 7” – that’s things like webmail.ns3.example.com cpanel.ns3.example.com etc etc), and we can see it’s setup the proper HTTP DCV validation with a different filename ( xyzxyz.txt ) which does actually correspond to the md5 hash of the certificate request. And we see it’s been submitted to the cPStore, but there is a processing delay.
Server can access it, but what about the certificate issuer?
Ah yes – so far, we’ve just fixed the preflight system allowing the server to check itself – but not allowed any third parties/services access. So which IP addresses do we need to allow/whitelist? Well, as soon as the “Order submitted” came up, cPanel’s cPStore passed the request to Sectigo to process and they made a request back. Let’s have a look in the access log to see if it was blocked.
# grep 401 /usr/local/cpanel/logs/access_log 91.199.212.132 - - [xxxxx] "-" 401 0 "-" "-" "-" "-" 80
Oh – an IP address we don’t recognise as one of ours. Now, it could be Sectigo or it could just be someone else randomly testing if we have a website. Let’s do an lookup to check.
Whois Lookups with Team Cymru
So how can we verify that we want to trust that IP address? Whilst we could just use rDNS (reverse DNS) to check:
# host 91.199.212.132 132.212.199.91.in-addr.arpa domain name pointer secure.trust-provider.net.
that doesn’t really help much (who are trust-provider.net? There’s no website at that URL and rDNS is reasonably easy to fake).
Instead, we’ll use WHOIS with the Team Cyrmu WHOIS service to lookup the “Autonomous System Number” (ASN) details which will be a lot more authoritative (as it’ll list who “owns” that entire netblock and the size of it). First we’ll need to install (using yum), the whois command but then we are away!
# yum install whois # whois -h whois.cymru.com "-v 91.199.212.132" Warning: RIPE flags used with a traditional server. AS | IP | BGP Prefix | CC | Registry | Allocated | AS Name 48447 | 91.199.212.132 | 91.199.212.0/24 | GB | ripencc | 2008-02-22 | SECTIGO, GB
We can see that the IP address 91.199.212.132 – according to the RIPE NCC Registry – belongs to AS48447 who are “Sectigo, GB”. And as we know cPanel Inc uses Sectigo – we’ve got the right company! (btw Sectigo used to be known as Comodo CA).
Finding Sectigo’s netblocks using PWhois
Whilst we could just allow the listed prefix/netblock of 91.199.212.0/24 access to the “cphttpd” service – there’s a good chance Sectigo have other netblocks allocated to them/their ASN. There’s another service we could utilise to find all their netblocks – PWhois.org (yes, we could have used PWhois for the initial IP to ASN lookup as well, but I didn’t so there 😉 ).
# whois -h whois.pwhois.org "routeview source-as=48447"
Origin-AS: 48447
Prefix | Create-Date | Modify-Date | Originated-Date | Next-Hop | AS-Path
*> 5.183.44.0/22 | Jul 18 2022 00:00:03 | Jul 18 2022 00:00:03 | Jul 09 2022 05:54:02 | 208.115.137.35 | 8220 1299 174 48447
*> 91.199.212.0/24 | Jul 18 2022 00:00:03 | Jul 18 2022 00:00:03 | Jul 09 2022 05:54:02 | 208.115.137.35 | 8220 1299 174 48447
*> 91.209.196.0/24 | Jul 18 2022 00:00:03 | Jul 18 2022 00:00:03 | Jul 09 2022 05:54:02 | 208.115.137.35 | 8220 1299 174 48447
*> 91.212.12.0/24 | Jul 18 2022 00:00:03 | Jul 18 2022 00:00:03 | Jul 09 2022 05:54:02 | 208.115.137.35 | 8220 1299 174 48447
Now we have all the netblocks allocated to AS48447 (at least all of those found by pwhois and it’s possible Sectigo have other ASN allocations with RIPE and even with other registries, but we’ll just ignore those facts for now 😉 ).
Allowing those netblocks
Now just edit /etc/hosts.allow again to add those ranges at the bottom (before the “ALL : ALL : deny” entry) allowing access to cphttpd (remembering to keep our server IP address (198.51.100.23) listed as otherwise the preflights will fail again): We’ll also add some comments ( starting with # ) just in case we forget why we added them.
cphttpd : 198.51.100.23 : allow # previous IP is our server IP address cphttpd : 5.183.44.0/22 : allow # previous address is Sectigo for web DCV access cphttpd : 91.199.212.0/24 : allow # previous address is Sectigo for web DCV access cphttpd : 91.209.196.0/24 : allow # previous address is Sectigo for web DCV access cphttpd : 91.212.12.0/24 : allow # previous address is Sectigo for web DCV access ALL : ALL : deny
Now just wait for the certificate to be issued (if, after an hour, it hasn’t been – then just rerun checkallsslcerts )
You can see the actual check from Sectigo if you monitor /usr/local/cpanel/logs/access_log :
# tail /usr/local/cpanel/logs/access_log ... 91.199.212.132 - - [xxxxx -0000] "GET /.well-known/pki-validation/xyzxyz.txt HTTP/1.1" 200 0 "-" "Sectigo DCV" "-" "-" 80
Allow the list cPanel Inc provide
For some reason, I didn’t see this handy page “What IP addresses do Sectigo DCV requests originate from?“on the cPanel knowledgebase before doing all this. So just change your /etc/hosts.allow file to have – at the bottom –
cphttpd : 198.51.100.23 : allow # previous IP is our server IP address cphttpd : 178.255.81.12 : allow # previous address is Sectigo for web DCV access cphttpd : 178.255.81.13 : allow # previous address is Sectigo for web DCV access cphttpd : 91.199.212.132 : allow # previous address is Sectigo for web DCV access cphttpd : 199.66.201.132 : allow # previous address is Sectigo for web DCV access cphttpd : 91.199.212.52 : allow # previous address is Sectigo for web DCV access cphttpd: [2a02:1788:400:1ce4::/64] : allow # previous address is Sectigo for web DCV access ALL : ALL : deny
Note the formatting of the IPv6 address.
All done!