Press "Enter" to skip to content

Category: Life: Work and Techy

ActivityPub for WordPress – How to fix ModSecurity to make it work

Like many people at the moment (due to Elon Musk’s purchase of Twitter), I’m moving from my nearly 14 year old Twitter account @rbairwell to Mastodon where I’m currently at . I was also pointed towards‘s WordPress plugin ActivityPub For WordPress which allows me to put my blog directly “on the Fediverse” and allow you to follow it at .

Symptoms / stuck on “Withdraw follow request”

However, after installing it the plugin and then trying to follow my blog, I just got a “Withdraw follow request” prompt in Mastodon – and, even after giving it a few minutes to account for server lag, my follow didn’t show up in WordPress->Users->Followers (Fediverse). If you want, you can just skip to the solution for root users .

Investigation / Mod Security Logs

My initial thought was that it was mod_security (a web-application firewall for the web site) which might be intercepting and blocking the request for security purposes. Turns out I was correct first time! Looking at my cPanel WHM's Security Center->ModSecurity Tools->Hits List, I found out that the requests were being blocked by rule 920420 of the OWASP Core Ruleset which was causing the following messages:

Rule id920420: Request content type is not allowed by policy
RequestPOST /wp-json/activitypub/1.0/users/3/inbox
Action DescriptionWarning.
JustificationMatch of “within %{tx.allowed_request_content_type}” against “TX:content_type” required.
Details of the mod_security hit

Searching the mod security audit log for the request URL using grep /wp-json/activitypub/ /var/log/apache2/modsec_audit.log gave me the “incident id/file location”: - - [xx/xxx/xxxx:xx:xx:xx +0000] "POST /wp-json/activitypub/1.0/users/3/inbox HTTP/1.1" 403 4077 "-" "-" Y2z65HnPJZ2EEJpVH6GcggAAAA8 "-" /xxxxx/20221110/20221110-1321/20221110-132140-Y2z65HnPJZ2EEJpVH6GcggAAAA8 0 5109 md5:39bb07d5be0cc904943570b3a39fddbc

looking at /var/log/apache2/modsec_audit/xxxxx/20221110/20221110-1321/20221110-132140-Y2z65HnPJZ2EEJpVH6GcggAAAA8 showed me

POST /wp-json/activitypub/1.0/users/3/inbox HTTP/1.1
Content-Type: application/activity+json
Apache-Error: [file "apache2_util.c"] [line 271] [level 3] [client] ModSecurity: Warning. Match of "within %{tx.allowed_request_content_type}" against "TX:content_type" required. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "956"] [id "920420"] [msg "Request content type is not allowed by policy"] [data "|application/activity+json|"] [severity "CRITICAL"] [ver "OWASP_CRS/3.3.2"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/255/153"] [tag "PCI/12.1"] [hostname ""] [uri "/wp-json/activitypub/1.0/users/3/inbox"] [unique_id "Y2z65HnPJZ2EEJpVH6GcggAAAA8"]

Showing me that the ActivityPub protocol makes requests using the Content-type of application/activity+json which isn’t normally allowed with the OWASP Core Ruleset (OWASP CRS/3.3.2).

So how to fix this?

If you do not have root accessto your server, you might just have the option to turn off mod_security totally for your domain which will restore access.

If you do have root access, you’ll be able to view rule 92040in either your control panel (WHM users->Security Center->ModSecurity Tools->Rules List) or in your server at the listed path ( /etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf ). However, you’ll find that it lists:

# In case Content-Type header can be parsed, check the mime-type against
# the policy defined in the 'allowed_request_content_type' variable.
# To change your policy, edit crs-setup.conf and activate rule 900220.
SecRule REQUEST_HEADERS:Content-Type "@rx ^[^;\s]+" \
    msg:'Request content type is not allowed by policy',\
    SecRule TX:content_type "!@within %{tx.allowed_request_content_type}" \

But not the list of actually content-types allowed. Whilst these are defined in rule 901162 (found by searching for “tx.allowed_request_content_type“), you shouldn’t really modify the “vendor supplied rules”.

it’s best to add your own rule 900220 which is within crs-setup.conf. But it’s not advisable to change that file (in /etc/apache2/conf.d/modsec_vendor_configs/OWASP3/crs-setup.conf on my cPanel server) on cPanel servers as it might get updated/changed by cPanel itself.

Adding the new mod security rule to allow application/activity+json

Therefore, I’ve just created a new rule within mod_security (again WHM->Security Center->ModSecurity Tools->Rules List->Add Rule ) to match it with the additional content type listed:

SecAction \
  setvar:'tx.allowed_request_content_type=|application/x-www-form-urlencoded| |multipart/form-data| |multipart/related| |text/xml| |application/xml| |application/soap+xml| |application/x-amf| |application/json| |application/octet-stream| |application/csp-report| |application/xss-auditor-report| |text/plain| |application/activity+json|'"

Note that the list of content types are separated by spaces, but are actually each enclosed by the pipe symbol – the pipe ( | ) isn’t the separator!

I deployed and restarted Apache and tried to follow myself again, and it all started working (and about 2 minutes after I posted this, it showed up in my timeline)

Hope it helps somebody else!

DNSSec signed Google Apps/G Suite Email

I’ve been using Google Apps, aka Google Workspace aka Google Suite (or just G Suite) for a while now and it’s annoyed me that I was getting “marked down” on e-mail security testers such as and the UK Government’s National Cyber Security Centre (NCSC) Check Your Email Security Service because Gmail for Business (G Suite) didn’t support DNSSEC (Domain Name System Security Extensions) signed MX hosts.

However, I’ve managed to find Google’s DNS Sec settings which – combined with other setups on my main domains – mean I get 4 green ticks from the NCSC, 97% from (I’m let down by Google’s support of old TLS and Ciphers settings and no DANE TLSA records) and all green (apart from DANE) on Hardenize : so nice strong secure email!

Read more

Google normally suggest you use the following MX (Mail Exchanger) records in your DNS settings if you use G Suite:

PriorityMail Server (MX Entry)
The normal suggest Google Suite Email Servers for Businesses

(The records can actually be in any order and the priority can be anything – but Google do recommend that is set as the “highest priority” which is actually 1)

However, after a bit of searching (using DuckDuckGo and not Google 😉 ), led me to a blog post by Nis Bornoe and Kura the following G Suite DNSSEC signed MX records:

Google’s DNSSEC Signed Mail Servers (MX Entry)
Google’s “hidden” DNS SEC Signed MX Records

These domains are hosted on Google owned Charleston Road Registry (CRR)’s .goog top level domain (not to be confused with their .google and .gle brand top-level domains: or the 98 other ones they applied for) and .goog domains can “only be registered to Google Inc and its affiliates” so you’ve got some confidence they are legitimate.

However, whilst myself and Nis and Kura do not seem to have had any problems using these IPv4 and IPv6 supported DNSSEC signed nameservers (and according to DNSlytics and WhoisXMLAPI there are over 930 domains currently using them), they are not officially supported or documented (from what we can find) and have been running since at least 2019 – so they should be reasonably safe to use.

The only “catch” may may be that, for some reason, they do NOT have a reverse DNS (Pointer aka PTR) record setup – which is actually only a problem if those mail servers are use for sending OUT email (not just receiving it) – however, many only testers do assume that your inbound and outbound mail servers are the same. I can confirm, via a test email, that outbound mail goes out via servers such as which are correctly configured.

Techy: Removing an rate limit block from Exim

During testing/investigation of an issue, I had to send “bad emails” to my Exim based mail server which after a number of attempts blocked me using the rate limit that had been configured:

451-The server has reached its limit for processing requests from your host.
451 Please try again later.

I didn’t want to wait for over an hour for the block to clear so I had to find out how to clear the block manually. On the mail server, I found the Exim data files in /var/spool/exim/db and using the exim_dumpdb tool to view the “ratelimit” block list:

# exim_dumpdb /var/spool/exim/ ratelimit

and then limiting it to the sending IP address (which I had checked via /var/log/exim_rejectlog ):

# exim_dumpdb /var/spool/exim/ ratelimit | grep 2001:db8:9:a::
28-Oct-2022 16:50:01.247104 rate:      1.877 key: 1h/per_mail/2001:db8:9:a::
28-Oct-2022 16:50:14.986765 rate:      1.937 key: 1h/per_conn//2001:db8:9:a::

Okay, that’s confirmed the block – but now to remove it. To do this, I had to use the “practically no documentation and no ui” tool exim_fixdb :

exim_fixdb /var/spool/exim/ ratelimit

It then just showed a prompt “>” with no instructions. But if the “key” was provided:

# exim_fixdb /var/spool/exim/ ratelimit
Modifying Exim hints database /var/spool/exim//db/ratelimit
> 1h/per_mail/2001:db8:9:a::
28-Oct-2022 16:50:01
0 time stamp: 28-Oct-2022 16:50:01
1 fract. time: .247104
2 sender rate: 1.877

Pressing “d” then deleted that record (and I repeated it for the other entry). Once I had finished, I just used “q” to quit exim_fixdb.

Hope it helps!

cPanel: Disabling cPanel Store Promotions

A few people have got a bit annoyed at the promotions/advertisements within their WHM webhosting control panel as developed by cPanel Inc (specifically, I saw the feature request “give server admins a way to turn off spam ads“).

Whilst cPanel partners (i.e. whoever your brought your cPanel licence from – usually your datacenter or web hosting provider) have the ability to toggle these cPanel Store Purchases from their cPanel Inc “manage2” interface, there is a way of disabling it yourself by “abusing” the internal testing code built into cPanel (specifically /usr/local/cpanel/Cpanel/Config/ which is called by /usr/local/cpanel/Cpanel/Whostmgr/ which, in turn, is called by /usr/local/cpanel/Whostmgr/Store/Product/ ):

cPanel DNSOnly Server Hostname SSL Renewal Issues – Fix

This has turned into a little “story” about how I resolved an issue with Sectigo SSL certificates failing HTTP DCV validation on a cPanel DNS Only server. It’s a bit long, but you can just jump to the conclusion at the end if you want to – or go to individual sections.

Read more

System Setup

  • I have a cPanel DNSOnly server setup running secondary DNS (gee, would have guessed by the product name? ? ) for a server. It’s setup to be a “read-only” server and not to send changes (synchronisation) to the main server.
  • Because that server should only offer DNS facilities to the “outside world”/general internet, it’s been locked down for security. Only a few select IP addresses can access SSH (on port 22) or WHM (on port 2087) – the entire internet can access UDP/TCP ports 53 (DNS) – but apart from that, there is no access whatsoever to the server.
  • The hostname of the server is in a domain name hosted by an external provider
  • The server has HSTS (HTTP Strict Transport Security) enabled which means – if the certificate has a fault – WHM will not be accessible. I’ll therefore do everything via the command line “just in case”.
  • For the explanation, the host name of the server will be and the server IP will be

What happened?

After about 65 days of the server being setup in the above configuration, I started receiving the following message (why 65 days? the SSL certificates are normally issued for 90 days and had cPanel’s contact notifications for “cPanel service SSL certificate warnings - This option indicates that a warning was generated while checking the cPanel service SSL certificates.” enabled (cPanel starts auto renewing when the certificate has 25 days or less left).

Subject: [ ] ? 1 service generated warnings while checking SSL certificates

The following cPanel service generated warnings from the checkallsslcerts script.
? cpanel
The system cannot install the fetched certificate (EXPIRES_SOON).
The system failed to acquire a signed certificate from the cPanel Store because of the following error: All HTTP and DNS DCV preflight checks failed!
This notice is the result of a request from “/usr/local/cpanel/bin/checkallsslcerts”.
The system generated this notice on xxxx.
“cPanel service SSL certificate warnings” notifications are currently configured to have an importance of “Medium”. You can change the importance or disable this type of notification in WHM’s Contact Manager at:
Do not reply to this automated message.

Why did this happen?

The cPanel Inc provided SSL certificate (provided via Sectigo) was due to expire within 30 days, but the server was unable to renew it because the security checks (Domain Control Validation – DCV) could not verify the server was in control of the hostname

It did try confirming it over DNS, but as the DNS was remotely/externally hosted, it could not make the necessary changes “in real time”. It then also tried confirming it over HTTP, but this failed as – being a DNS server – this server does not run a web server such as Apache or nginx.

First checkallsslcerts test: What is failing?

The first step was to try and see what was happening. This was done by running the cPanel checkallsslcerts command and seeing the output (some bits shortened):

# /usr/local/cpanel/bin/checkallsslcerts --allow-retry --verbose
The system will check for the certificate for the “cpanel” service.
The system will attempt to install a certificate for the “cpanel” service from the cPanel store.
Setting up HTTP DCV (/usr/local/apache/htdocs/.well-known/pki-validation/xxxxxxxx.txt) …
… complete.
Setting up DNS DCV for “” …
… complete.
Attempting DNS DCV preflight checks … DNS DCV preflight check failed; falling back to HTTP …
.... Attempting HTTP DCV preflight check …
The system queried for a temporary file at “”, but the web server responded with the following error: 401 (Access Denied). A DNS (Domain Name System) or web server misconfiguration may exist.
Undoing HTTP DCV setup …
… complete.
Undoing DNS DCV setup …
… complete.
[WARN] The system failed to acquire a signed certificate from the cPanel Store because of the following error: All HTTP and DNS DCV preflight checks failed!

As we can see, the failure happened within the “preflight” section (where the server checks itself that things are working before it makes a call out). Let’s see if we can work out why – we’ll ignore the DNS DCV method as the server can’t control the external domain name.

Checking the web path

The file cPanel creates for DCV preflight checks (and also for the actual certificate request) is removed at the end of the run (“Undoing HTTP DCV setup...“), so we’ll have to make a temporary file to test – the checkallsslcerts system used the folder /usr/local/apache/htdocs/.well-known/pki-validation , so let’s make a temporary file in there:

# touch /usr/local/apache/htdocs/.well-known/pki-validation/my-test-file.txt

and try accessing it via a web browser at . Oh. it worked. Odd… Well, not really – my office IP address is “allowed” to access nearly everything (the “old terminology” would be “whitelisted”).

Checking the web path from the server

But the server is complaining about the preflight – so let’s try the request from the server itself.

# curl -I
HTTP/1.0 401 Access Denied
Connection: close
Content-Type: text/html; charset="utf-8"
Date: xxxx
Cache-Control: no-cache, no-store, must-revalidate, private
Pragma: no-cache
X-Error-Message: Access Denied
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
Content-Length: 5121

So the URL is blocked from being accessed by the server. But by what? Because we know that the path is /usr/local/apache/htdocs/ (and not /var/www/html/ or /home/xxxx/public_html ), and the fact there is no “Server:” identifier included in the returned headers, there stands a high chance of this being served from cPanel itself with it’s built in “mini webserver” (which is normally used to serve the WHM, cPanel and Webmail UIs).

We can check this by using cPanel’s access log:

# grep 401 /usr/local/cpanel/logs/access_log - [xxxxxx -0000] "-" 401 0 "-" "-" "-" "-" 80

So, yes, it is cPanel blocking it. We can confirm this by using cPanel’s error log:

# grep /usr/local/cpanel/logs/error_log
Dropping connection from because of host access control at line 4128.

Brilliant – and we’ve got a clue of what we need to allow – host access control.

Host Access Control

As you may have seen from the “Host Access Control” web pages in WHM, you are required to enter a “TCP Wrapper service name” (or “ALL“), an IP address (or “ALL“) and whether you are wanting to “allow” or “deny” access. This file is stored as /etc/hosts.allow .

But the problem we have is we don’t know which service name to enter. I tried “ALL” and that worked, but that would allow access to everything – which I didn’t want. I then tried all the “known service names”:

  • snmp (“SNMP Service”)
  • sshd (“SSH Service”)
  • pop3 (“Pop3 Service Daemon”)
  • domain (“DNS Services”)
  • auth (“Ident Service”)
  • cpaneld (“cPanel Service Daemon”)
  • postgresql (“PostgreSQL Service”)
  • smtp (“SMTP Service Daemon”)
  • whostmgrd (“Web Host Manager Service Daemon”)
  • cpdavd (“WebDav/WebDisk Service Daemon”)
  • telnet (“Telnet Service”)
  • ftp (“Ftp Server”)
  • mysql (“MySQL Server”)
  • imap (“Imap Service Daemon”)
  • webmaild (“WebMail Service Daemon”)

(taken from a WHM “Host Access Control” page I could access and examining the source),

but none of them worked. So which service is it?

Host Access Control – Finding The Service

In /etc/hosts.allow I added the following line near the bottom of the file before the “ALL : ALL : deny” line:

ALL : : spawn /bin/echo `date` %c %d >> /root/test.txt

This causes the TCP wrapper service to launch a process (spawn) echoing the current date, the IP address (%c) and the service name (%d) to the file /root/test.txt [ see the Softpanorama page for TCP Wrappers under “Shell Commands”). I then re-tried the curl command from the server and looked at /root/test.txt

# cat /root/test.txt
xxxxx UTC 2022 cphttpd

Bingo – it’s the “cphttpd” service which isn’t listed in WHM: no wonder that “ALL” would allow it but none of the other services did. Now let’s remodify /etc/hosts.allow to allow that service for our IP address:

cphttpd: : allow

Repeat the curl test a third (or forth?) time and bingo!

# curl -I
HTTP/1.1 200 OK

Success – we’ve got the server accessing itself!

We can now delete the test file:

# rm /usr/local/apache/htdocs/.well-known/pki-validation/my-test-file.txt

Second checkallsslcerts test – Checking preflight

Let’s try checkallsslcerts again and see how far we get:

# /usr/local/cpanel/bin/checkallsslcerts --allow-retry --verbose
The system will check for the certificate for the “cpanel” service.
The system will attempt to install a certificate for the “cpanel” service from the cPanel store.
Setting up HTTP DCV (/usr/local/apache/htdocs/.well-known/pki-validation/xxxxxxxx.txt) …
… complete.
Setting up DNS DCV for “” …
… complete.
Attempting DNS DCV preflight checks …
.... Attempting HTTP DCV preflight check …
        … success!
Succeeded domains: 1
Failed domains: 7
Undoing HTTP DCV setup …
        … complete.
Undoing DNS DCV setup …
        … complete.
Setting up HTTP DCV (/usr/local/apache/htdocs/.well-known/pki-validation/xyzxyz.txt) …
        … complete.
Setting up DNS DCV for “” …
        … complete.

Requesting certificate from cPStore …
        Order submitted. (Order item ID: 12345)
... [short wait] ...
The cPanel Store is processing the hostname certificate request.
The system will check the cPanel Store again in an hour to see if the cPanel Store issued the certificate.

Brilliant – we passed the preflight check with “success!”, had 1 succeeded domain (don’t worry about the “failed domains: 7” – that’s things like etc etc), and we can see it’s setup the proper HTTP DCV validation with a different filename ( xyzxyz.txt ) which does actually correspond to the md5 hash of the certificate request. And we see it’s been submitted to the cPStore, but there is a processing delay.

Server can access it, but what about the certificate issuer?

Ah yes – so far, we’ve just fixed the preflight system allowing the server to check itself – but not allowed any third parties/services access. So which IP addresses do we need to allow/whitelist? Well, as soon as the “Order submitted” came up, cPanel’s cPStore passed the request to Sectigo to process and they made a request back. Let’s have a look in the access log to see if it was blocked.

# grep 401 /usr/local/cpanel/logs/access_log - - [xxxxx] "-" 401 0 "-" "-" "-" "-" 80

Oh – an IP address we don’t recognise as one of ours. Now, it could be Sectigo or it could just be someone else randomly testing if we have a website. Let’s do an lookup to check.

Whois Lookups with Team Cymru

So how can we verify that we want to trust that IP address? Whilst we could just use rDNS (reverse DNS) to check:

# host domain name pointer

that doesn’t really help much (who are There’s no website at that URL and rDNS is reasonably easy to fake).

Instead, we’ll use WHOIS with the Team Cyrmu WHOIS service to lookup the “Autonomous System Number” (ASN) details which will be a lot more authoritative (as it’ll list who “owns” that entire netblock and the size of it). First we’ll need to install (using yum), the whois command but then we are away!

# yum install whois
# whois -h "-v"
Warning: RIPE flags used with a traditional server.
AS      | IP               | BGP Prefix          | CC | Registry | Allocated  | AS Name
48447   |   |     | GB | ripencc  | 2008-02-22 | SECTIGO, GB

We can see that the IP address – according to the RIPE NCC Registry – belongs to AS48447 who are “Sectigo, GB”. And as we know cPanel Inc uses Sectigo – we’ve got the right company! (btw Sectigo used to be known as Comodo CA).

Finding Sectigo’s netblocks using PWhois

Whilst we could just allow the listed prefix/netblock of access to the “cphttpd” service – there’s a good chance Sectigo have other netblocks allocated to them/their ASN. There’s another service we could utilise to find all their netblocks – (yes, we could have used PWhois for the initial IP to ASN lookup as well, but I didn’t so there 😉 ).

# whois -h "routeview source-as=48447"
Origin-AS: 48447
    Prefix            | Create-Date              | Modify-Date              | Originated-Date          | Next-Hop        | AS-Path
*> |     Jul 18 2022 00:00:03 |     Jul 18 2022 00:00:03 |     Jul 09 2022 05:54:02 | | 8220 1299 174 48447
*> |     Jul 18 2022 00:00:03 |     Jul 18 2022 00:00:03 |     Jul 09 2022 05:54:02 | | 8220 1299 174 48447
*> |     Jul 18 2022 00:00:03 |     Jul 18 2022 00:00:03 |     Jul 09 2022 05:54:02 | | 8220 1299 174 48447
*> |     Jul 18 2022 00:00:03 |     Jul 18 2022 00:00:03 |     Jul 09 2022 05:54:02 | | 8220 1299 174 48447

Now we have all the netblocks allocated to AS48447 (at least all of those found by pwhois and it’s possible Sectigo have other ASN allocations with RIPE and even with other registries, but we’ll just ignore those facts for now 😉 ).

Allowing those netblocks

Now just edit /etc/hosts.allow again to add those ranges at the bottom (before the “ALL : ALL : deny” entry) allowing access to cphttpd (remembering to keep our server IP address ( listed as otherwise the preflights will fail again): We’ll also add some comments ( starting with # ) just in case we forget why we added them.

cphttpd : : allow
# previous IP is our server IP address
cphttpd : : allow
# previous address is Sectigo for web DCV access
cphttpd : : allow
# previous address is Sectigo for web DCV access
cphttpd : : allow
# previous address is Sectigo for web DCV access
cphttpd : : allow
# previous address is Sectigo for web DCV access
ALL : ALL : deny

Now just wait for the certificate to be issued (if, after an hour, it hasn’t been – then just rerun checkallsslcerts )

You can see the actual check from Sectigo if you monitor /usr/local/cpanel/logs/access_log :

# tail /usr/local/cpanel/logs/access_log
... - - [xxxxx -0000] "GET /.well-known/pki-validation/xyzxyz.txt HTTP/1.1" 200 0 "-" "Sectigo DCV" "-" "-" 80

Allow the list cPanel Inc provide

For some reason, I didn’t see this handy page “What IP addresses do Sectigo DCV requests originate from?“on the cPanel knowledgebase before doing all this. So just change your /etc/hosts.allow file to have – at the bottom –

cphttpd : : allow
# previous IP is our server IP address
cphttpd : : allow
# previous address is Sectigo for web DCV access
cphttpd : : allow
# previous address is Sectigo for web DCV access
cphttpd : : allow
# previous address is Sectigo for web DCV access
cphttpd : : allow
# previous address is Sectigo for web DCV access
cphttpd : : allow
# previous address is Sectigo for web DCV access
cphttpd: [2a02:1788:400:1ce4::/64] : allow
# previous address is Sectigo for web DCV access
ALL : ALL : deny

Note the formatting of the IPv6 address.

All done!