Random ramblings and ravings of Richy B
In an effort to prove to myself that I am actually trying to do work this last month, I’m making a note of all the bugs in 3rd party software I find.
Today is a bug reported to cPanel Inc on the 6th June 2022 under their tracking request ID 94453274 affecting outbound emails sent from servers running their WHM/cPanel software to web hosting customers whose account has an notification.
This has turned into a little “story” about how I resolved an issue with Sectigo SSL certificates failing HTTP DCV validation on a cPanel DNS Only server. It’s a bit long, but you can just jump to the conclusion at the end if you want to – or go to individual sections.
Read more: cPanel DNSOnly Server Hostname SSL Renewal Issues – FixAfter about 65 days of the server being setup in the above configuration, I started receiving the following message (why 65 days? the SSL certificates are normally issued for 90 days and had cPanel’s contact notifications for “cPanel service SSL certificate warnings - This option indicates that a warning was generated while checking the cPanel service SSL certificates.” enabled (cPanel starts auto renewing when the certificate has 25 days or less left).
Subject: [ns3.example.com ] ? 1 service generated warnings while checking SSL certificates
The following cPanel service generated warnings from the checkallsslcerts script.
? cpanel
The system cannot install the fetched certificate (EXPIRES_SOON).
The system failed to acquire a signed certificate from the cPanel Store because of the following error: All HTTP and DNS DCV preflight checks failed!
This notice is the result of a request from “/usr/local/cpanel/bin/checkallsslcerts”. The system generated this notice on xxxx.
“cPanel service SSL certificate warnings” notifications are currently configured to have an importance of “Medium”. You can change the importance or disable this type of notification in WHM’s Contact Manager at: https://ns3.example.com:2087/scripts2/editcontact?event=SSL::CheckAllCertsWarnings
Do not reply to this automated message.
The cPanel Inc provided SSL certificate (provided via Sectigo) was due to expire within 30 days, but the server was unable to renew it because the security checks (Domain Control Validation – DCV) could not verify the server was in control of the hostname ns3.example.com
It did try confirming it over DNS, but as the DNS was remotely/externally hosted, it could not make the necessary changes “in real time”. It then also tried confirming it over HTTP, but this failed as – being a DNS server – this server does not run a web server such as Apache or nginx.
The first step was to try and see what was happening. This was done by running the cPanel checkallsslcerts command and seeing the output (some bits shortened):
# /usr/local/cpanel/bin/checkallsslcerts --allow-retry --verbose The system will check for the certificate for the “cpanel” service. ... The system will attempt to install a certificate for the “cpanel” service from the cPanel store. Setting up HTTP DCV (/usr/local/apache/htdocs/.well-known/pki-validation/xxxxxxxx.txt) … … complete. Setting up DNS DCV for “ns3.example.com” … … complete. Attempting DNS DCV preflight checks … ns3.example.com: DNS DCV preflight check failed; falling back to HTTP … .... ns3.example.com: Attempting HTTP DCV preflight check … The system queried for a temporary file at “http://ns3.example.com/.well-known/pki-validation/xxxxxxxx.txt”, but the web server responded with the following error: 401 (Access Denied). A DNS (Domain Name System) or web server misconfiguration may exist. Undoing HTTP DCV setup … … complete. Undoing DNS DCV setup … … complete. [WARN] The system failed to acquire a signed certificate from the cPanel Store because of the following error: All HTTP and DNS DCV preflight checks failed!
As we can see, the failure happened within the “preflight” section (where the server checks itself that things are working before it makes a call out). Let’s see if we can work out why – we’ll ignore the DNS DCV method as the server can’t control the external domain name.
The file cPanel creates for DCV preflight checks (and also for the actual certificate request) is removed at the end of the run (“Undoing HTTP DCV setup...“), so we’ll have to make a temporary file to test – the checkallsslcerts system used the folder /usr/local/apache/htdocs/.well-known/pki-validation , so let’s make a temporary file in there:
# touch /usr/local/apache/htdocs/.well-known/pki-validation/my-test-file.txt
and try accessing it via a web browser at http://ns3.example.com/.well-known/pki-validation/my-test-file.txt . Oh. it worked. Odd… Well, not really – my office IP address is “allowed” to access nearly everything (the “old terminology” would be “whitelisted”).
But the server is complaining about the preflight – so let’s try the request from the server itself.
# curl -I http://ns3.example.com/.well-known/pki-validation/test.txt HTTP/1.0 401 Access Denied Connection: close Content-Type: text/html; charset="utf-8" Date: xxxx Cache-Control: no-cache, no-store, must-revalidate, private Pragma: no-cache X-Error-Message: Access Denied X-Frame-Options: SAMEORIGIN X-Content-Type-Options: nosniff Content-Length: 5121
So the URL is blocked from being accessed by the server. But by what? Because we know that the path is /usr/local/apache/htdocs/ (and not /var/www/html/ or /home/xxxx/public_html ), and the fact there is no “Server:” identifier included in the returned headers, there stands a high chance of this being served from cPanel itself with it’s built in “mini webserver” (which is normally used to serve the WHM, cPanel and Webmail UIs).
We can check this by using cPanel’s access log:
# grep 401 /usr/local/cpanel/logs/access_log 198.51.100.23- - [xxxxxx -0000] "-" 401 0 "-" "-" "-" "-" 80
So, yes, it is cPanel blocking it. We can confirm this by using cPanel’s error log:
# grep 198.51.100.23 /usr/local/cpanel/logs/error_log Dropping connection from 198.51.100.23 because of host access control at cpsrvd.pl line 4128.
Brilliant – and we’ve got a clue of what we need to allow – host access control.
As you may have seen from the “Host Access Control” web pages in WHM, you are required to enter a “TCP Wrapper service name” (or “ALL“), an IP address (or “ALL“) and whether you are wanting to “allow” or “deny” access. This file is stored as /etc/hosts.allow .
But the problem we have is we don’t know which service name to enter. I tried “ALL” and that worked, but that would allow access to everything – which I didn’t want. I then tried all the “known service names”:
(taken from a WHM “Host Access Control” page I could access and examining the source),
but none of them worked. So which service is it?
In /etc/hosts.allow I added the following line near the bottom of the file before the “ALL : ALL : deny” line:
ALL : 198.51.100.23 : spawn /bin/echo `date` %c %d >> /root/test.txt
This causes the TCP wrapper service to launch a process (spawn) echoing the current date, the IP address (%c) and the service name (%d) to the file /root/test.txt [ see the Softpanorama page for TCP Wrappers under “Shell Commands”). I then re-tried the curl command from the server and looked at /root/test.txt
# cat /root/test.txt xxxxx UTC 2022 198.51.100.23 cphttpd
Bingo – it’s the “cphttpd” service which isn’t listed in WHM: no wonder that “ALL” would allow it but none of the other services did. Now let’s remodify /etc/hosts.allow to allow that service for our 198.51.100.23 IP address:
cphttpd: 198.51.100.23 : allow
Repeat the curl test a third (or forth?) time and bingo!
# curl -I http://ns3.example.com/.well-known/pki-validation/test.txt HTTP/1.1 200 OK
Success – we’ve got the server accessing itself!
We can now delete the test file:
# rm /usr/local/apache/htdocs/.well-known/pki-validation/my-test-file.txt
Let’s try checkallsslcerts again and see how far we get:
# /usr/local/cpanel/bin/checkallsslcerts --allow-retry --verbose
The system will check for the certificate for the “cpanel” service.
...
The system will attempt to install a certificate for the “cpanel” service from the cPanel store.
Setting up HTTP DCV (/usr/local/apache/htdocs/.well-known/pki-validation/xxxxxxxx.txt) …
… complete.
Setting up DNS DCV for “ns3.example.com” …
… complete.
Attempting DNS DCV preflight checks …
....
ns3.example.com: Attempting HTTP DCV preflight check …
… success!
...
Succeeded domains: 1
Failed domains: 7
Undoing HTTP DCV setup …
… complete.
Undoing DNS DCV setup …
… complete.
Setting up HTTP DCV (/usr/local/apache/htdocs/.well-known/pki-validation/xyzxyz.txt) …
… complete.
Setting up DNS DCV for “ns3.example.com” …
… complete.
Requesting certificate from cPStore …
Order submitted. (Order item ID: 12345)
... [short wait] ...
The cPanel Store is processing the hostname certificate request.
The system will check the cPanel Store again in an hour to see if the cPanel Store issued the certificate.
Brilliant – we passed the preflight check with “success!”, had 1 succeeded domain (don’t worry about the “failed domains: 7” – that’s things like webmail.ns3.example.com cpanel.ns3.example.com etc etc), and we can see it’s setup the proper HTTP DCV validation with a different filename ( xyzxyz.txt ) which does actually correspond to the md5 hash of the certificate request. And we see it’s been submitted to the cPStore, but there is a processing delay.
Ah yes – so far, we’ve just fixed the preflight system allowing the server to check itself – but not allowed any third parties/services access. So which IP addresses do we need to allow/whitelist? Well, as soon as the “Order submitted” came up, cPanel’s cPStore passed the request to Sectigo to process and they made a request back. Let’s have a look in the access log to see if it was blocked.
# grep 401 /usr/local/cpanel/logs/access_log 91.199.212.132 - - [xxxxx] "-" 401 0 "-" "-" "-" "-" 80
Oh – an IP address we don’t recognise as one of ours. Now, it could be Sectigo or it could just be someone else randomly testing if we have a website. Let’s do an lookup to check.
So how can we verify that we want to trust that IP address? Whilst we could just use rDNS (reverse DNS) to check:
# host 91.199.212.132 132.212.199.91.in-addr.arpa domain name pointer secure.trust-provider.net.
that doesn’t really help much (who are trust-provider.net? There’s no website at that URL and rDNS is reasonably easy to fake).
Instead, we’ll use WHOIS with the Team Cyrmu WHOIS service to lookup the “Autonomous System Number” (ASN) details which will be a lot more authoritative (as it’ll list who “owns” that entire netblock and the size of it). First we’ll need to install (using yum), the whois command but then we are away!
# yum install whois # whois -h whois.cymru.com "-v 91.199.212.132" Warning: RIPE flags used with a traditional server. AS | IP | BGP Prefix | CC | Registry | Allocated | AS Name 48447 | 91.199.212.132 | 91.199.212.0/24 | GB | ripencc | 2008-02-22 | SECTIGO, GB
We can see that the IP address 91.199.212.132 – according to the RIPE NCC Registry – belongs to AS48447 who are “Sectigo, GB”. And as we know cPanel Inc uses Sectigo – we’ve got the right company! (btw Sectigo used to be known as Comodo CA).
Whilst we could just allow the listed prefix/netblock of 91.199.212.0/24 access to the “cphttpd” service – there’s a good chance Sectigo have other netblocks allocated to them/their ASN. There’s another service we could utilise to find all their netblocks – PWhois.org (yes, we could have used PWhois for the initial IP to ASN lookup as well, but I didn’t so there 😉 ).
# whois -h whois.pwhois.org "routeview source-as=48447"
Origin-AS: 48447
Prefix | Create-Date | Modify-Date | Originated-Date | Next-Hop | AS-Path
*> 5.183.44.0/22 | Jul 18 2022 00:00:03 | Jul 18 2022 00:00:03 | Jul 09 2022 05:54:02 | 208.115.137.35 | 8220 1299 174 48447
*> 91.199.212.0/24 | Jul 18 2022 00:00:03 | Jul 18 2022 00:00:03 | Jul 09 2022 05:54:02 | 208.115.137.35 | 8220 1299 174 48447
*> 91.209.196.0/24 | Jul 18 2022 00:00:03 | Jul 18 2022 00:00:03 | Jul 09 2022 05:54:02 | 208.115.137.35 | 8220 1299 174 48447
*> 91.212.12.0/24 | Jul 18 2022 00:00:03 | Jul 18 2022 00:00:03 | Jul 09 2022 05:54:02 | 208.115.137.35 | 8220 1299 174 48447
Now we have all the netblocks allocated to AS48447 (at least all of those found by pwhois and it’s possible Sectigo have other ASN allocations with RIPE and even with other registries, but we’ll just ignore those facts for now 😉 ).
Now just edit /etc/hosts.allow again to add those ranges at the bottom (before the “ALL : ALL : deny” entry) allowing access to cphttpd (remembering to keep our server IP address (198.51.100.23) listed as otherwise the preflights will fail again): We’ll also add some comments ( starting with # ) just in case we forget why we added them.
cphttpd : 198.51.100.23 : allow # previous IP is our server IP address cphttpd : 5.183.44.0/22 : allow # previous address is Sectigo for web DCV access cphttpd : 91.199.212.0/24 : allow # previous address is Sectigo for web DCV access cphttpd : 91.209.196.0/24 : allow # previous address is Sectigo for web DCV access cphttpd : 91.212.12.0/24 : allow # previous address is Sectigo for web DCV access ALL : ALL : deny
Now just wait for the certificate to be issued (if, after an hour, it hasn’t been – then just rerun checkallsslcerts )
You can see the actual check from Sectigo if you monitor /usr/local/cpanel/logs/access_log :
# tail /usr/local/cpanel/logs/access_log ... 91.199.212.132 - - [xxxxx -0000] "GET /.well-known/pki-validation/xyzxyz.txt HTTP/1.1" 200 0 "-" "Sectigo DCV" "-" "-" 80
For some reason, I didn’t see this handy page “What IP addresses do Sectigo DCV requests originate from?“on the cPanel knowledgebase before doing all this. So just change your /etc/hosts.allow file to have – at the bottom –
cphttpd : 198.51.100.23 : allow # previous IP is our server IP address cphttpd : 178.255.81.12 : allow # previous address is Sectigo for web DCV access cphttpd : 178.255.81.13 : allow # previous address is Sectigo for web DCV access cphttpd : 91.199.212.132 : allow # previous address is Sectigo for web DCV access cphttpd : 199.66.201.132 : allow # previous address is Sectigo for web DCV access cphttpd : 91.199.212.52 : allow # previous address is Sectigo for web DCV access cphttpd: [2a02:1788:400:1ce4::/64] : allow # previous address is Sectigo for web DCV access ALL : ALL : deny
Note the formatting of the IPv6 address.
All done!
In an effort to prove to myself that I am actually trying to do work this month, I’m making a note of all the bugs in 3rd party software I find.
Today is a bug reported to cPanel Inc on the 4th June 2022 under their tracking request ID 94452913 affecting their WordPress Toolkit cPanel module which has a problem with slashes in plugin names.
We tend to operate RedHat Enterprise/CentOS based cPanel web hosting accounts for ourselves and our customers and for security, they are configured running PHP under suPHP via CGI/FastCGI. However, for server monitoring, we wanted to be able to use NewRelic for monitoring the sites on an individual level but without the customers having to add newrelic.appname to each of their applications – and we didn’t want each site to have to have its own php.ini, and under suPHP PHP ignores any php_value settings in the Apache configuration. So what to do?
Install New Relic
Well, first of all install New Relic according to your server architecture, then run “newrelic-install” and select all. Now we move on to the cPanel specific setup.
Getting PHP to notice the NewRelic PHP Module
In WHM (:2087), select “PHP Configuration Editor” and then “Advanced mode”. Search for Core “extension” and change the list from something like:
/usr/lib/php/extensions/no-debug-non-zts-20090626/apc.so, pdo.so, pdo_sqlite.so, sqlite.so, pdo_mysql.so, timezonedb.so, uploadprogress.so
to:
/usr/lib/php/extensions/no-debug-non-zts-20090626/apc.so, pdo.so, pdo_sqlite.so, sqlite.so, pdo_mysql.so, timezonedb.so, uploadprogress.so, newrelic.so
Restart Apache (via either WHM or /scripts/restartsrv_apache on the command line) and check everything is still working.
Configuring the newrelic.appname settings – Apache
From from command line/shell, cd into /var/cpanel/templates/apache2_2 (or whichever version of Apache you are running) and copy vhost.default to vhost.local (cp vhost.default vhost.local) if there isn’t already a vhost.local file. Now open that file up in your favourite text editor (for simplicities sake, I’m using nano : nano -w /var/cpanel/templates/apache2_2/vhost.local).
Near the top of the file it will say something like:
<VirtualHost[% FOREACH ipblock IN vhost.ips %] [% ipblock.ip %]:[% ipblock.port %][% END %]>
ServerName [% wildcard_safe(vhost.servername) %]
[% IF vhost.serveralias_array.size -%]
[% FOREACH alias IN vhost.serveralias_array -%]
ServerAlias [% alias %]
[% END -%]
Add a suitable “SetEnv newrelic_appname” line. For example, I want all of the sites to be recorded against the main virtual host’s server name, so I would use:
<VirtualHost[% FOREACH ipblock IN vhost.ips %] [% ipblock.ip %]:[% ipblock.port %][% END %]>
ServerName [% wildcard_safe(vhost.servername) %]
SetEnv newrelic_appname [% wildcard_safe(vhost.servername) %]
[% IF vhost.serveralias_array.size -%]
[% FOREACH alias IN vhost.serveralias_array -%]
ServerAlias [% alias %]
[% END -%]
You could also use the user’s username SetEnv newrelic_appname [% % vhost.user %] if you wanted to.
Use the Distiller to check everything looks good:
/usr/local/cpanel/bin/apache_conf_distiller --update
and if it does, rebuild the Apache configuration:
/usr/local/cpanel/bin/build_apache_conf
and again restart Apache and test.
Now, there is an environment variable called “newrelic_appname” available for usage. But we’re not using it yet: so we move onto:
Configuring the newrelic.appname settings – PHP
Using a <?php phpinfo(); ?> script find where your php.ini file – ours was in /usr/local/lib/php.ini . Open it open in your text editor again (nano -w /usr/local/lib/php.ini) and search for the newrelic.appname setting which should appear (Press Ctrl+W and then type appname in nano). You should see something like:
; Setting: newrelic.appname
; Type : string
; Scope : per-directory
; Default: “PHP Application”
; Info : Sets the name of the application that metrics will be reported into.
; This can in fact be a list of up to 3 application names, each of
; which must be separated by a semi-colon. The first name in any such
; list is considered the ‘primary’ application name and must be unique
; for each account / license key.
;
;newrelic.appname = “PHP Application”;
; Beginning with version 3.0 of the agent, the daemon can be automatically
; started by the agent. There is no need to start the daemon before starting
Remove the colon and the start of the line and change it to read from ${newrelic_appname} for example:
; for each account / license key.
;
newrelic.appname = “${newrelic_appname}”;
; Beginning with version 3.0 of the agent, the daemon can be automatically
This will read in the newrelic_appname environment variable upon run and dynamically replace it in the PHP settings. Restart Apache again (just to be safe) and check a few sites are working. Within a couple of minutes, you should see the records appear in NewRelic.
All done! Have fun!
If, like myself, you have recently upgraded the cPanel control panel system on your Red Hat Enterprise Linux 5.2 server and received the error message “Error: Package system can not be repaired automatically” when upgrading Apache, then it’s probably caused by a problem with either the RedHat Package Manager (RPM) or YUM. To fix it, first ensure that no RPM or yum updates are running:
ps auxwww | grep yum
ps auxwww | grep rpm
If any are running, find out why and kill -9 them if they are zombie processes. Now you just need to rebuild the RPM database:
rm /var/lib/rpm/__db.* -rf
rpm --rebuilddb
This process may take some time (between 1 and 30 minutes depending on your server speed and the number of packages installed). Once it’s completed, you should be able to upgrade without problems.