Press "Enter" to skip to content

Category: Net: Techy: Linux

Posts concerning Linux

ActivityPub for WordPress – How to fix ModSecurity to make it work

Like many people at the moment (due to Elon Musk’s purchase of Twitter), I’m moving from my nearly 14 year old Twitter account @rbairwell to Mastodon where I’m currently at @rbairwell@mastodon.org.uk . I was also pointed towards @pfefferle@mastodon.social‘s WordPress plugin ActivityPub For WordPress which allows me to put my blog directly “on the Fediverse” and allow you to follow it at @richyb@blog.rac.me.uk .

Symptoms / stuck on “Withdraw follow request”

However, after installing it the plugin and then trying to follow my blog, I just got a “Withdraw follow request” prompt in Mastodon – and, even after giving it a few minutes to account for server lag, my follow didn’t show up in WordPress->Users->Followers (Fediverse). If you want, you can just skip to the solution for root users .

Investigation / Mod Security Logs

My initial thought was that it was mod_security (a web-application firewall for the web site) which might be intercepting and blocking the request for security purposes. Turns out I was correct first time! Looking at my cPanel WHM's Security Center->ModSecurity Tools->Hits List, I found out that the requests were being blocked by rule 920420 of the OWASP Core Ruleset which was causing the following messages:

FieldData
Rule id920420: Request content type is not allowed by policy
SeverityCritical
Status403
RequestPOST /wp-json/activitypub/1.0/users/3/inbox
Action DescriptionWarning.
JustificationMatch of “within %{tx.allowed_request_content_type}” against “TX:content_type” required.
Details of the mod_security hit

Searching the mod security audit log for the request URL using grep /wp-json/activitypub/ /var/log/apache2/modsec_audit.log gave me the “incident id/file location”:

blog.rac.me.uk xxx.xxx.xxx.xxx - - [xx/xxx/xxxx:xx:xx:xx +0000] "POST /wp-json/activitypub/1.0/users/3/inbox HTTP/1.1" 403 4077 "-" "-" Y2z65HnPJZ2EEJpVH6GcggAAAA8 "-" /xxxxx/20221110/20221110-1321/20221110-132140-Y2z65HnPJZ2EEJpVH6GcggAAAA8 0 5109 md5:39bb07d5be0cc904943570b3a39fddbc

looking at /var/log/apache2/modsec_audit/xxxxx/20221110/20221110-1321/20221110-132140-Y2z65HnPJZ2EEJpVH6GcggAAAA8 showed me

...
--daee5752-B--
POST /wp-json/activitypub/1.0/users/3/inbox HTTP/1.1
Host: blog.rac.me.uk
...
Content-Type: application/activity+json
...
--daee5752-H--
...
Apache-Error: [file "apache2_util.c"] [line 271] [level 3] [client xxx.xxx.xxx.xxx] ModSecurity: Warning. Match of "within %{tx.allowed_request_content_type}" against "TX:content_type" required. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "956"] [id "920420"] [msg "Request content type is not allowed by policy"] [data "|application/activity+json|"] [severity "CRITICAL"] [ver "OWASP_CRS/3.3.2"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/255/153"] [tag "PCI/12.1"] [hostname "blog.rac.me.uk"] [uri "/wp-json/activitypub/1.0/users/3/inbox"] [unique_id "Y2z65HnPJZ2EEJpVH6GcggAAAA8"]

Showing me that the ActivityPub protocol makes requests using the Content-type of application/activity+json which isn’t normally allowed with the OWASP Core Ruleset (OWASP CRS/3.3.2).

So how to fix this?

If you do not have root accessto your server, you might just have the option to turn off mod_security totally for your domain which will restore access.

If you do have root access, you’ll be able to view rule 92040in either your control panel (WHM users->Security Center->ModSecurity Tools->Rules List) or in your server at the listed path ( /etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf ). However, you’ll find that it lists:

# In case Content-Type header can be parsed, check the mime-type against
# the policy defined in the 'allowed_request_content_type' variable.
# To change your policy, edit crs-setup.conf and activate rule 900220.
SecRule REQUEST_HEADERS:Content-Type "@rx ^[^;\s]+" \
    "id:1,\
    phase:2,\
    block,\
    capture,\
    t:none,\
    msg:'Request content type is not allowed by policy',\
    logdata:'%{MATCHED_VAR}',\
    tag:'application-multi',\
    tag:'language-multi',\
    tag:'platform-multi',\
    tag:'attack-protocol',\
    tag:'paranoia-level/1',\
    tag:'OWASP_CRS',\
    tag:'capec/1000/255/153',\
    tag:'PCI/12.1',\
    ver:'OWASP_CRS/3.3.2',\
    severity:'CRITICAL',\
    setvar:'tx.content_type=|%{tx.0}|',\
    chain"
    SecRule TX:content_type "!@within %{tx.allowed_request_content_type}" \
        "t:lowercase,\
        setvar:'tx.anomaly_score_pl1=+%{tx.critical_anomaly_score}'"

But not the list of actually content-types allowed. Whilst these are defined in rule 901162 (found by searching for “tx.allowed_request_content_type“), you shouldn’t really modify the “vendor supplied rules”.

it’s best to add your own rule 900220 which is within crs-setup.conf. But it’s not advisable to change that file (in /etc/apache2/conf.d/modsec_vendor_configs/OWASP3/crs-setup.conf on my cPanel server) on cPanel servers as it might get updated/changed by cPanel itself.

Adding the new mod security rule to allow application/activity+json

Therefore, I’ve just created a new rule within mod_security (again WHM->Security Center->ModSecurity Tools->Rules List->Add Rule ) to match it with the additional content type listed:

SecAction \
 "id:900220,\
  phase:1,\
  nolog,\
  pass,\
  t:none,\
  setvar:'tx.allowed_request_content_type=|application/x-www-form-urlencoded| |multipart/form-data| |multipart/related| |text/xml| |application/xml| |application/soap+xml| |application/x-amf| |application/json| |application/octet-stream| |application/csp-report| |application/xss-auditor-report| |text/plain| |application/activity+json|'"

Note that the list of content types are separated by spaces, but are actually each enclosed by the pipe symbol – the pipe ( | ) isn’t the separator!

I deployed and restarted Apache and tried to follow myself again, and it all started working (and about 2 minutes after I posted this, it showed up in my timeline)

Hope it helps somebody else!

Techy: Removing an rate limit block from Exim

During testing/investigation of an issue, I had to send “bad emails” to my Exim based mail server which after a number of attempts blocked me using the rate limit that had been configured:

451-The server has reached its limit for processing requests from your host.
451 Please try again later.

I didn’t want to wait for over an hour for the block to clear so I had to find out how to clear the block manually. On the mail server, I found the Exim data files in /var/spool/exim/db and using the exim_dumpdb tool to view the “ratelimit” block list:

# exim_dumpdb /var/spool/exim/ ratelimit

and then limiting it to the sending IP address (which I had checked via /var/log/exim_rejectlog ):

# exim_dumpdb /var/spool/exim/ ratelimit | grep 2001:db8:9:a::
28-Oct-2022 16:50:01.247104 rate:      1.877 key: 1h/per_mail/2001:db8:9:a::
28-Oct-2022 16:50:14.986765 rate:      1.937 key: 1h/per_conn//2001:db8:9:a::

Okay, that’s confirmed the block – but now to remove it. To do this, I had to use the “practically no documentation and no ui” tool exim_fixdb :

exim_fixdb /var/spool/exim/ ratelimit

It then just showed a prompt “>” with no instructions. But if the “key” was provided:

# exim_fixdb /var/spool/exim/ ratelimit
Modifying Exim hints database /var/spool/exim//db/ratelimit
> 1h/per_mail/2001:db8:9:a::
28-Oct-2022 16:50:01
0 time stamp: 28-Oct-2022 16:50:01
1 fract. time: .247104
2 sender rate: 1.877

Pressing “d” then deleted that record (and I repeated it for the other entry). Once I had finished, I just used “q” to quit exim_fixdb.

Hope it helps!

Techy: Changing Python Shebang path from absolute to env

I’m just working in some Python code, and quite a lot of it has the “shebang” line for the Python interpreter set as an absolute path (such as /usr/bin/python) which won’t work for me as I need the scripts to use whichever version of Python I have configured in my “environment”: and the ideal way to do this is to call is using /usr/bin/env python – so how do I change these…

Techy: Nixstats And Cloudflare – Allowing IPs

What is NixStats?

I’m currently trying the NixStats (affiliate link) monitoring tool for servers and websites – mainly because it’s cheap (starts at $0/month for 5 monitors and 1 server, $9.95/month for 10 servers and 25 web monitors) and partially because it’s owned by WebPros. Not heard of WebPros? Me neither until I went digging and found out that not only do they now own NixStats but also cPanel Inc and Plesk (both “rivals” in the web hosting control panel world) – along with a few others(!)

The Problem – Cloudflare’s security being triggered

Anyway – one of the sites I am monitoring using Nixstats is under the Cloudflare system – and because of the frequent monitoring, it may “trip up” Cloudflare’s security features so I needed to “allow list” (aka “white list”) the IP addresses used by NixStats. Luckily, they do provide a list at https://nixstats.com/whitelist.php and https://nixstats.com/whitelist.php?v6 (I’ve got no idea how often these are updated though) – but you can’t simply import these into Cloudflare due to two issues:

  1. One of the IP addresses (in the first list) has a trailing tab character which Cloudflare just rejects as being an invalid IP address. Took me a while to find that!
  2. The IPv6 addresses are not in a format that Cloudflare accepts – they only accept CIDR’d IPv6 address on a /64 or smaller (and if you are not that techy, you are forgiven for not understanding that!).

The solution

So I’ve put together a downloadable CSV file listing all those IP addresses in the “correct format” for you to easily import into Cloudflare’s lists (Cloudflare don’t have an export option 🙁 ).

You can download the file from here but I give no guarantee of accuracy etc etc.

But where in Cloudflare do I use the file?

It’s a two stage operation – first you’ve got to create the list and then you’ve got to setup the firewall rules.

Creating the IP Address List in Cloudflare

  1. Log into your Cloudflare account and get onto the main account screen (i.e. where it lists your individual sites)
  2. On the left hand menu near the bottom, there is an option labelled “Manage Account” – expand that.
  3. Under “Manage Account”, select “Configurations”
  4. On the “Configurations” screen, select “Lists”
  5. Select “Create new list” (free Cloudflare accounts can only create a single list by the way)
  6. Create the list with a name such as “nixstats” and a “Content type” of “IP Addresses”
  7. Edit the list and select “Add items”
  8. Select “Upload CSV” and upload the CSV of IP addresses ,
  9. Click “Add to list” and it should load them into the list.

Setting Cloudflare to use the list

  1. Log into Cloudflare (or, if you are already logged in, click the Cloudflare logo in the top left to return to home)
  2. Select the account of the domain name you are monitoring
  3. In the left hand menu, expand “Security”
  4. Under “Security”, select “WAF” (Web Application Firewall)
  5. Under “Firewall rules” (free accounts get 5 active firewall rules), select “Create firewall rule”
  6. Set a “Rule name” of something like “nixstats”
  7. Under the “When incoming requests match…” heading use the following settings
    • Field: “IP Source Address”
    • Operator: “is in list”
    • Value: “nixstats” (i.e. the list you created above)
  8. Under “Then…” select “Allow”
  9. Click “Deploy firewall rule” and it should take affect.

You’ll need to do this for each account you are monitoring using nixstats.

New Relic, cPanel, Apache and FastCGI PHP

We tend to operate RedHat Enterprise/CentOS based cPanel web hosting accounts for ourselves and our customers and for security, they are configured running PHP under suPHP via CGI/FastCGI. However, for server monitoring, we wanted to be able to use NewRelic for monitoring the sites on an individual level but without the customers having to add newrelic.appname to each of their applications – and we didn’t want each site to have to have its own php.ini, and under suPHP PHP ignores any php_value settings in the Apache configuration. So what to do?

Install New Relic
Well, first of all install New Relic according to your server architecture, then run “newrelic-install” and select all. Now we move on to the cPanel specific setup.

Getting PHP to notice the NewRelic PHP Module
In WHM (:2087), select “PHP Configuration Editor” and then “Advanced mode”. Search for Core “extension” and change the list from something like:
/usr/lib/php/extensions/no-debug-non-zts-20090626/apc.so, pdo.so, pdo_sqlite.so, sqlite.so, pdo_mysql.so, timezonedb.so, uploadprogress.so
to:
/usr/lib/php/extensions/no-debug-non-zts-20090626/apc.so, pdo.so, pdo_sqlite.so, sqlite.so, pdo_mysql.so, timezonedb.so, uploadprogress.so, newrelic.so

Restart Apache (via either WHM or /scripts/restartsrv_apache on the command line) and check everything is still working.

Configuring the newrelic.appname settings – Apache
From from command line/shell, cd into /var/cpanel/templates/apache2_2 (or whichever version of Apache you are running) and copy vhost.default to vhost.local (cp vhost.default vhost.local) if there isn’t already a vhost.local file. Now open that file up in your favourite text editor (for simplicities sake, I’m using nano : nano -w /var/cpanel/templates/apache2_2/vhost.local).

Near the top of the file it will say something like:

<VirtualHost[% FOREACH ipblock IN vhost.ips %] [% ipblock.ip %]:[% ipblock.port %][% END %]>
ServerName [% wildcard_safe(vhost.servername) %]
[% IF vhost.serveralias_array.size -%]
[% FOREACH alias IN vhost.serveralias_array -%]
ServerAlias [% alias %]
[% END -%]

Add a suitable “SetEnv newrelic_appname” line. For example, I want all of the sites to be recorded against the main virtual host’s server name, so I would use:

<VirtualHost[% FOREACH ipblock IN vhost.ips %] [% ipblock.ip %]:[% ipblock.port %][% END %]>
ServerName [% wildcard_safe(vhost.servername) %]
SetEnv newrelic_appname [% wildcard_safe(vhost.servername) %]
[% IF vhost.serveralias_array.size -%]
[% FOREACH alias IN vhost.serveralias_array -%]
ServerAlias [% alias %]
[% END -%]

You could also use the user’s username SetEnv newrelic_appname [% % vhost.user %] if you wanted to.

Use the Distiller to check everything looks good:
/usr/local/cpanel/bin/apache_conf_distiller --update
and if it does, rebuild the Apache configuration:
/usr/local/cpanel/bin/build_apache_conf
and again restart Apache and test.

Now, there is an environment variable called “newrelic_appname” available for usage. But we’re not using it yet: so we move onto:

Configuring the newrelic.appname settings – PHP
Using a <?php phpinfo(); ?> script find where your php.ini file – ours was in /usr/local/lib/php.ini . Open it open in your text editor again (nano -w /usr/local/lib/php.ini) and search for the newrelic.appname setting which should appear (Press Ctrl+W and then type appname in nano). You should see something like:

; Setting: newrelic.appname
; Type : string
; Scope : per-directory
; Default: “PHP Application”
; Info : Sets the name of the application that metrics will be reported into.
; This can in fact be a list of up to 3 application names, each of
; which must be separated by a semi-colon. The first name in any such
; list is considered the ‘primary’ application name and must be unique
; for each account / license key.
;
;newrelic.appname = “PHP Application”

;
; Beginning with version 3.0 of the agent, the daemon can be automatically
; started by the agent. There is no need to start the daemon before starting

Remove the colon and the start of the line and change it to read from ${newrelic_appname} for example:

; for each account / license key.
;
newrelic.appname = “${newrelic_appname}”

;
; Beginning with version 3.0 of the agent, the daemon can be automatically

This will read in the newrelic_appname environment variable upon run and dynamically replace it in the PHP settings. Restart Apache again (just to be safe) and check a few sites are working. Within a couple of minutes, you should see the records appear in NewRelic.

All done! Have fun!