Press "Enter" to skip to content

Month: July 2022

cPanel DNSOnly Server Hostname SSL Renewal Issues – Fix

This has turned into a little “story” about how I resolved an issue with Sectigo SSL certificates failing HTTP DCV validation on a cPanel DNS Only server. It’s a bit long, but you can just jump to the conclusion at the end if you want to – or go to individual sections.

Read more

System Setup

  • I have a cPanel DNSOnly server setup running secondary DNS (gee, would have guessed by the product name? ? ) for a server. It’s setup to be a “read-only” server and not to send changes (synchronisation) to the main server.
  • Because that server should only offer DNS facilities to the “outside world”/general internet, it’s been locked down for security. Only a few select IP addresses can access SSH (on port 22) or WHM (on port 2087) – the entire internet can access UDP/TCP ports 53 (DNS) – but apart from that, there is no access whatsoever to the server.
  • The hostname of the server is in a domain name hosted by an external provider
  • The server has HSTS (HTTP Strict Transport Security) enabled which means – if the certificate has a fault – WHM will not be accessible. I’ll therefore do everything via the command line “just in case”.
  • For the explanation, the host name of the server will be ns3.example.com and the server IP will be 198.51.100.23

What happened?

After about 65 days of the server being setup in the above configuration, I started receiving the following message (why 65 days? the SSL certificates are normally issued for 90 days and had cPanel’s contact notifications for “cPanel service SSL certificate warnings - This option indicates that a warning was generated while checking the cPanel service SSL certificates.” enabled (cPanel starts auto renewing when the certificate has 25 days or less left).

Subject: [ns3.example.com ] ? 1 service generated warnings while checking SSL certificates

The following cPanel service generated warnings from the checkallsslcerts script.
? cpanel
The system cannot install the fetched certificate (EXPIRES_SOON).
The system failed to acquire a signed certificate from the cPanel Store because of the following error: All HTTP and DNS DCV preflight checks failed!
This notice is the result of a request from “/usr/local/cpanel/bin/checkallsslcerts”.
The system generated this notice on xxxx.
“cPanel service SSL certificate warnings” notifications are currently configured to have an importance of “Medium”. You can change the importance or disable this type of notification in WHM’s Contact Manager at: https://ns3.example.com:2087/scripts2/editcontact?event=SSL::CheckAllCertsWarnings
Do not reply to this automated message.

Why did this happen?

The cPanel Inc provided SSL certificate (provided via Sectigo) was due to expire within 30 days, but the server was unable to renew it because the security checks (Domain Control Validation – DCV) could not verify the server was in control of the hostname ns3.example.com

It did try confirming it over DNS, but as the DNS was remotely/externally hosted, it could not make the necessary changes “in real time”. It then also tried confirming it over HTTP, but this failed as – being a DNS server – this server does not run a web server such as Apache or nginx.

First checkallsslcerts test: What is failing?

The first step was to try and see what was happening. This was done by running the cPanel checkallsslcerts command and seeing the output (some bits shortened):

# /usr/local/cpanel/bin/checkallsslcerts --allow-retry --verbose
The system will check for the certificate for the “cpanel” service.
...
The system will attempt to install a certificate for the “cpanel” service from the cPanel store.
Setting up HTTP DCV (/usr/local/apache/htdocs/.well-known/pki-validation/xxxxxxxx.txt) …
… complete.
Setting up DNS DCV for “ns3.example.com” …
… complete.
Attempting DNS DCV preflight checks …
ns3.example.com: DNS DCV preflight check failed; falling back to HTTP …
....
ns3.example.com: Attempting HTTP DCV preflight check …
The system queried for a temporary file at “http://ns3.example.com/.well-known/pki-validation/xxxxxxxx.txt”, but the web server responded with the following error: 401 (Access Denied). A DNS (Domain Name System) or web server misconfiguration may exist.
Undoing HTTP DCV setup …
… complete.
Undoing DNS DCV setup …
… complete.
[WARN] The system failed to acquire a signed certificate from the cPanel Store because of the following error: All HTTP and DNS DCV preflight checks failed!

As we can see, the failure happened within the “preflight” section (where the server checks itself that things are working before it makes a call out). Let’s see if we can work out why – we’ll ignore the DNS DCV method as the server can’t control the external domain name.

Checking the web path

The file cPanel creates for DCV preflight checks (and also for the actual certificate request) is removed at the end of the run (“Undoing HTTP DCV setup...“), so we’ll have to make a temporary file to test – the checkallsslcerts system used the folder /usr/local/apache/htdocs/.well-known/pki-validation , so let’s make a temporary file in there:

# touch /usr/local/apache/htdocs/.well-known/pki-validation/my-test-file.txt

and try accessing it via a web browser at http://ns3.example.com/.well-known/pki-validation/my-test-file.txt . Oh. it worked. Odd… Well, not really – my office IP address is “allowed” to access nearly everything (the “old terminology” would be “whitelisted”).

Checking the web path from the server

But the server is complaining about the preflight – so let’s try the request from the server itself.

# curl -I http://ns3.example.com/.well-known/pki-validation/test.txt
HTTP/1.0 401 Access Denied
Connection: close
Content-Type: text/html; charset="utf-8"
Date: xxxx
Cache-Control: no-cache, no-store, must-revalidate, private
Pragma: no-cache
X-Error-Message: Access Denied
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
Content-Length: 5121

So the URL is blocked from being accessed by the server. But by what? Because we know that the path is /usr/local/apache/htdocs/ (and not /var/www/html/ or /home/xxxx/public_html ), and the fact there is no “Server:” identifier included in the returned headers, there stands a high chance of this being served from cPanel itself with it’s built in “mini webserver” (which is normally used to serve the WHM, cPanel and Webmail UIs).

We can check this by using cPanel’s access log:

# grep 401 /usr/local/cpanel/logs/access_log
198.51.100.23- - [xxxxxx -0000] "-" 401 0 "-" "-" "-" "-" 80

So, yes, it is cPanel blocking it. We can confirm this by using cPanel’s error log:

# grep 198.51.100.23 /usr/local/cpanel/logs/error_log
Dropping connection from 198.51.100.23 because of host access control at cpsrvd.pl line 4128.

Brilliant – and we’ve got a clue of what we need to allow – host access control.

Host Access Control

As you may have seen from the “Host Access Control” web pages in WHM, you are required to enter a “TCP Wrapper service name” (or “ALL“), an IP address (or “ALL“) and whether you are wanting to “allow” or “deny” access. This file is stored as /etc/hosts.allow .

But the problem we have is we don’t know which service name to enter. I tried “ALL” and that worked, but that would allow access to everything – which I didn’t want. I then tried all the “known service names”:

  • snmp (“SNMP Service”)
  • sshd (“SSH Service”)
  • pop3 (“Pop3 Service Daemon”)
  • domain (“DNS Services”)
  • auth (“Ident Service”)
  • cpaneld (“cPanel Service Daemon”)
  • postgresql (“PostgreSQL Service”)
  • smtp (“SMTP Service Daemon”)
  • whostmgrd (“Web Host Manager Service Daemon”)
  • cpdavd (“WebDav/WebDisk Service Daemon”)
  • telnet (“Telnet Service”)
  • ftp (“Ftp Server”)
  • mysql (“MySQL Server”)
  • imap (“Imap Service Daemon”)
  • webmaild (“WebMail Service Daemon”)

(taken from a WHM “Host Access Control” page I could access and examining the source),

but none of them worked. So which service is it?

Host Access Control – Finding The Service

In /etc/hosts.allow I added the following line near the bottom of the file before the “ALL : ALL : deny” line:

ALL : 198.51.100.23 : spawn /bin/echo `date` %c %d >> /root/test.txt

This causes the TCP wrapper service to launch a process (spawn) echoing the current date, the IP address (%c) and the service name (%d) to the file /root/test.txt [ see the Softpanorama page for TCP Wrappers under “Shell Commands”). I then re-tried the curl command from the server and looked at /root/test.txt

# cat /root/test.txt
xxxxx UTC 2022 198.51.100.23 cphttpd

Bingo – it’s the “cphttpd” service which isn’t listed in WHM: no wonder that “ALL” would allow it but none of the other services did. Now let’s remodify /etc/hosts.allow to allow that service for our 198.51.100.23 IP address:

cphttpd: 198.51.100.23 : allow

Repeat the curl test a third (or forth?) time and bingo!

# curl -I http://ns3.example.com/.well-known/pki-validation/test.txt
HTTP/1.1 200 OK

Success – we’ve got the server accessing itself!

We can now delete the test file:

# rm /usr/local/apache/htdocs/.well-known/pki-validation/my-test-file.txt

Second checkallsslcerts test – Checking preflight

Let’s try checkallsslcerts again and see how far we get:

# /usr/local/cpanel/bin/checkallsslcerts --allow-retry --verbose
The system will check for the certificate for the “cpanel” service.
...
The system will attempt to install a certificate for the “cpanel” service from the cPanel store.
Setting up HTTP DCV (/usr/local/apache/htdocs/.well-known/pki-validation/xxxxxxxx.txt) …
… complete.
Setting up DNS DCV for “ns3.example.com” …
… complete.
Attempting DNS DCV preflight checks …
....
ns3.example.com: Attempting HTTP DCV preflight check …
        … success!
...
Succeeded domains: 1
Failed domains: 7
Undoing HTTP DCV setup …
        … complete.
Undoing DNS DCV setup …
        … complete.
Setting up HTTP DCV (/usr/local/apache/htdocs/.well-known/pki-validation/xyzxyz.txt) …
        … complete.
Setting up DNS DCV for “ns3.example.com” …
        … complete.

Requesting certificate from cPStore …
        Order submitted. (Order item ID: 12345)
... [short wait] ...
The cPanel Store is processing the hostname certificate request.
The system will check the cPanel Store again in an hour to see if the cPanel Store issued the certificate.

Brilliant – we passed the preflight check with “success!”, had 1 succeeded domain (don’t worry about the “failed domains: 7” – that’s things like webmail.ns3.example.com cpanel.ns3.example.com etc etc), and we can see it’s setup the proper HTTP DCV validation with a different filename ( xyzxyz.txt ) which does actually correspond to the md5 hash of the certificate request. And we see it’s been submitted to the cPStore, but there is a processing delay.

Server can access it, but what about the certificate issuer?

Ah yes – so far, we’ve just fixed the preflight system allowing the server to check itself – but not allowed any third parties/services access. So which IP addresses do we need to allow/whitelist? Well, as soon as the “Order submitted” came up, cPanel’s cPStore passed the request to Sectigo to process and they made a request back. Let’s have a look in the access log to see if it was blocked.

# grep 401 /usr/local/cpanel/logs/access_log
91.199.212.132 - - [xxxxx] "-" 401 0 "-" "-" "-" "-" 80

Oh – an IP address we don’t recognise as one of ours. Now, it could be Sectigo or it could just be someone else randomly testing if we have a website. Let’s do an lookup to check.

Whois Lookups with Team Cymru

So how can we verify that we want to trust that IP address? Whilst we could just use rDNS (reverse DNS) to check:

# host 91.199.212.132
132.212.199.91.in-addr.arpa domain name pointer secure.trust-provider.net.

that doesn’t really help much (who are trust-provider.net? There’s no website at that URL and rDNS is reasonably easy to fake).

Instead, we’ll use WHOIS with the Team Cyrmu WHOIS service to lookup the “Autonomous System Number” (ASN) details which will be a lot more authoritative (as it’ll list who “owns” that entire netblock and the size of it). First we’ll need to install (using yum), the whois command but then we are away!

# yum install whois
# whois -h whois.cymru.com "-v 91.199.212.132"
Warning: RIPE flags used with a traditional server.
AS      | IP               | BGP Prefix          | CC | Registry | Allocated  | AS Name
48447   | 91.199.212.132   | 91.199.212.0/24     | GB | ripencc  | 2008-02-22 | SECTIGO, GB

We can see that the IP address 91.199.212.132 – according to the RIPE NCC Registry – belongs to AS48447 who are “Sectigo, GB”. And as we know cPanel Inc uses Sectigo – we’ve got the right company! (btw Sectigo used to be known as Comodo CA).

Finding Sectigo’s netblocks using PWhois

Whilst we could just allow the listed prefix/netblock of 91.199.212.0/24 access to the “cphttpd” service – there’s a good chance Sectigo have other netblocks allocated to them/their ASN. There’s another service we could utilise to find all their netblocks – PWhois.org (yes, we could have used PWhois for the initial IP to ASN lookup as well, but I didn’t so there 😉 ).

# whois -h whois.pwhois.org "routeview source-as=48447"
Origin-AS: 48447
    Prefix            | Create-Date              | Modify-Date              | Originated-Date          | Next-Hop        | AS-Path
*>      5.183.44.0/22 |     Jul 18 2022 00:00:03 |     Jul 18 2022 00:00:03 |     Jul 09 2022 05:54:02 |  208.115.137.35 | 8220 1299 174 48447
*>    91.199.212.0/24 |     Jul 18 2022 00:00:03 |     Jul 18 2022 00:00:03 |     Jul 09 2022 05:54:02 |  208.115.137.35 | 8220 1299 174 48447
*>    91.209.196.0/24 |     Jul 18 2022 00:00:03 |     Jul 18 2022 00:00:03 |     Jul 09 2022 05:54:02 |  208.115.137.35 | 8220 1299 174 48447
*>     91.212.12.0/24 |     Jul 18 2022 00:00:03 |     Jul 18 2022 00:00:03 |     Jul 09 2022 05:54:02 |  208.115.137.35 | 8220 1299 174 48447

Now we have all the netblocks allocated to AS48447 (at least all of those found by pwhois and it’s possible Sectigo have other ASN allocations with RIPE and even with other registries, but we’ll just ignore those facts for now 😉 ).

Allowing those netblocks

Now just edit /etc/hosts.allow again to add those ranges at the bottom (before the “ALL : ALL : deny” entry) allowing access to cphttpd (remembering to keep our server IP address (198.51.100.23) listed as otherwise the preflights will fail again): We’ll also add some comments ( starting with # ) just in case we forget why we added them.

cphttpd : 198.51.100.23 : allow
# previous IP is our server IP address
cphttpd : 5.183.44.0/22 : allow
# previous address is Sectigo for web DCV access
cphttpd : 91.199.212.0/24 : allow
# previous address is Sectigo for web DCV access
cphttpd : 91.209.196.0/24 : allow
# previous address is Sectigo for web DCV access
cphttpd : 91.212.12.0/24 : allow
# previous address is Sectigo for web DCV access
ALL : ALL : deny

Now just wait for the certificate to be issued (if, after an hour, it hasn’t been – then just rerun checkallsslcerts )

You can see the actual check from Sectigo if you monitor /usr/local/cpanel/logs/access_log :

# tail /usr/local/cpanel/logs/access_log
...
91.199.212.132 - - [xxxxx -0000] "GET /.well-known/pki-validation/xyzxyz.txt HTTP/1.1" 200 0 "-" "Sectigo DCV" "-" "-" 80

Allow the list cPanel Inc provide

For some reason, I didn’t see this handy page “What IP addresses do Sectigo DCV requests originate from?“on the cPanel knowledgebase before doing all this. So just change your /etc/hosts.allow file to have – at the bottom –

cphttpd : 198.51.100.23 : allow
# previous IP is our server IP address
cphttpd : 178.255.81.12 : allow
# previous address is Sectigo for web DCV access
cphttpd : 178.255.81.13 : allow
# previous address is Sectigo for web DCV access
cphttpd : 91.199.212.132 : allow
# previous address is Sectigo for web DCV access
cphttpd : 199.66.201.132 : allow
# previous address is Sectigo for web DCV access
cphttpd : 91.199.212.52 : allow
# previous address is Sectigo for web DCV access
cphttpd: [2a02:1788:400:1ce4::/64] : allow
# previous address is Sectigo for web DCV access
ALL : ALL : deny

Note the formatting of the IPv6 address.

All done!

Techy: Microsoft PowerShell and Linux Commands

I’m finding myself mix and matching between GNU/Linux and Microsoft PowerShell quite a bit at the moment and which most Linuxy/Linuxesque (?) commands work fine under PowerShell, some are a little bit different.

Here’s a few which may be handy to know (the only order is that which I entered them). I’ve also added a couple of Windows commands where they are slightly easier to remember/shorter than PowerShell – and, me being me, included links to the appropriate documentation where possible.

Linux CommandPowerShell CommandDescription
which <commandname>
Example:
which composer
GNU Logo GNU “which”
Get-Command <commandname>
Example:
Get-Command composer
Powershell Logo PowerShell “Getting Environment Variables”
Find the location of an executable called <commandname>
In PowerShell, Get-Command can be referred to as just “gcm” so the example would become “gcm composer
printenv
GNU Logo GNU “printenv”
Get-Item -Path Env:
Powershell Logo PowerShell “Getting Environment Variables”
List all environment variables.
PowerShell also supports using “ls” to list environment variables – such as “ls env:
<variablename>="<value>"
Example:
HELLO="you guys"
Set-Item -Path Env:<variablename> "<value>"
Example:
Set-Item -Path Env:HELLO "you guys"
Powershell Logo PowerShell: “Change the value of an environment variable”
Sets an environment variable called <variablename> to the text <value>

PowerShell also supports using “$env:<variablename>="<value>” ” – such as ” $env:HELLO="you guys"
echo $<variablename>
Example:
echo $PATH
GNU Logo GNU “echo”
Get-ChildItem -Path Env:<variablename>
Example:
Get-ChildItem -Path Env:PATH
Powershell Logo PowerShell “Get A Selected Environment Variable”
Show the value of an environment variable called <variablename>
PowerShell also supports using “ls” to list environment variables – such as “ls env:PATH

echo <text>
Example:
echo "Hello there!"

GNU Logo GNU “echo”
Write-Output <text>
Example:
Write-Output "Hello there!"
Powershell Logo PowerShell “Write-Output”
Displays/prints a <text> string to the terminal
grep "<string>" <file>
Example:
grep "sort code" document.txt
GNU Logo GNU “grep”
Select-String "<string>" <file>
Example:
Select-String “sort code” document.txt
Powershell Logo PowerShell “Select-String”
Search a file <file> for the given text <string> using a regular expression

grep "<string>" <file>
Example:
grep "sort code" document.txt
GNU Logo GNU “grep”
findstr /R "<string>" <file>
Example:
findstr /R "sort code" document.txt
Windows LogoWindows Command: “findstr”
Search a file <file> for the given text <string> using a regular expression

ls | grep "<string>"
Example:
ls | grep "\.html"
GNU Logo GNU “ls”
ls | Out-String -Stream | Select-String "<string>"
Example:
ls | Out-String -Stream | Select-String “\.html”
Powershell Logo PowerShell “Select- String: Convert Pipline Objects”
Search a directory listing for a filename containing <string> using a regular expression
ls | grep "<string>"
Example:
ls | grep "\.html"
GNU Logo GNU “ls”
dir | findstr /R "<string>"
Example:
dir | findstr /R "\.html"
Windows LogoWindows Command: “findstr”
Search a directory listing for a filename containing <string> using a regular expression.
wget <url> --output-document <filename>
Example:
wget https://h.tld/f.gif --output-document o.gif
GNU Logo GNU “wget”
Invoke-WebRequest <url> -OutFile <filename>
Example:
Invoke-WebRequest https://h.tld/f.gif -OutFile o.gif
Powershell Logo PowerShell “Invoke-WebRequest”
Save a file at a URL <url> as a local file called <filename>
\
(backslash)
Example:
a \
b
GNU Logo GNU “The Backslash Character”

(backtick)
Example:
a
`
b
Powershell Logo PowerShell “…Line Continuation in Code…”
Allows a command to be split across multiple lines using the multiline separator/line continuation character.
&&
Example:
echo "A" && echo "B"
GNU Logo GNU “Lists
&&
Example:
Write-Output "A "&& Write-Output "B"
Powershell Logo PowerShell “Pipeline Chain Operators”
Command chaining using pipeline operators/list operators – if the condition on the left is true/passes, then continue.
rm <file>
Example:
rm test.tmp
GNU Logo GNU “rm”
Remove-Item <file>
Example:
Remove-Item test.tmp
Powershell Logo PowerShell “Remove-Item”
Deletes a file called <file>
unzip <file>
Example:
unzip myfile.zip
Expand-Archive <file>
Example:
Expand-Archive myfile.zip
Powershell Logo PowerShell “Expand-Archive”
Extracts a .zip archive file
uname
Example:
uname -nrmo
GNU Logo GNU “uname”
(note, the output order of uname in GNU/Linux cannot be altered)
Get-ComputerInfo
Example:
Get-ComputerInfo -Property CsDNSHostName, OsVersion, OsArchitecture, OsName | ConvertTo-Json
Powershell Logo PowerShell “Get- ComputerInfo”
Powershell Logo PowerShell “ConvertTo-Json”
Gets basic information about the system. For example:
Computer’s “hostname”
[GNU:n PS: CsDNSHostName]
OS or kernel release
[GNU: r PS:OsVersion]
Machine/Processor type
[GNU:m PS:OsArchitecture]
Operating system (OS) name
[GNU:o PS: OsName]
These commands were testing on a Debian bullseye Linux virtual machine and using Microsoft PowerShell 7.2 on Windows 10.

Microsoft PowerShell – don’t you mean Windows PowerShell?

Nope.

If you have a Windows 11, 10, 8.1, 8.0 or 7 machine, it would have come “with default” with Windows PowerShell – however, Microsoft has replaced that with a new “multiplatform” version which – in their infinite wisdom – have decided to call Microsoft PowerShell but haven’t “forcibly upgraded” people (you can upgrade yourself via their “Installing PowerShell on Windows” guide).

Microsoft do have a webpage about the differences between the PowerShells if you want to read: but if you use the command line a lot, it might be worth looking at Windows Terminal as well.

Which version of PowerShell do I have?

If you type/copy:

(Get-Host).Version

into the shell, you’ll get:

  • A “Major” version number of 7 or above = (new) Microsoft PowerShell
  • A “Major” version number of 5 or lower = (old) Windows PowerShell
  • A “Major” version number of 6 = I don’t think this actually exists and therefore should be used as the “cut-off” point between the two.
  • “.Version was unexpected at this time” = Probably using Windows Command Prompt
  • “-ash: syntax error: unexpected word” = You are in a shell of a Linux machine!
  • “bash: syntax error unexpected token `.Version’ = Linux again!

Real Life Example

As an example, here’s a single line Linux “command script” which uses curl to fetch a page from https://api.example/latest , parse it for the first occurrence of the text:

"tag_name":"v..."

(where v…. is any sequence of characters starting with “v” and ending with a quote mark) and store that as $VERSION then pass that to curl again to download that specific version zip file from https://example.com/v${VERSION}.zip as “example.zip” (replacing ${VERSION} with the extracted version number), unzip it and then delete the downloaded .zip file.

And, yes, I’ve colour coded the appropriate sections so you can see which command “maps” to which other command.

Linux Example

VERSION=$(
 curl --silent "https://api.example.com/latest" | \
 grep '"tag_name":' | \
 sed -E 's/.*"v([^"]+)".*/\1/' \
) && curl -L https://example.com/v${VERSION}.zip -o example.zip \
&& unzip example.zip && rm example.zip

Windows PowerShell 7 Example

if (((Invoke-WebRequest "https://api.example.com/latest").Content) -match '"tag_name":"v(?<ver>[^"]+)"') {
$VERSION=$Matches.ver &&
Invoke-WebRequest "https://example.com/v${VERSION}.zip" -OutFile example.zip &&
Expand-Archive example.zip && Remove-Item example.zip }