Traditionally, the load average of our Alma Linux server running our Apache/PHP-FPM/MariaDB website has been 1-3. We’ve recently experienced some good press and it appears our server is getting more traffic and the load is sustaining at 4-6 with burst above 10. We use Cloudflare, when I turn on “Under Attack Mode”, which essentially puts up the page to verify visitors are human, it immediately subsides to back to traditional levels. I’ve been watching Apache server status and see the requests being processed jumps from 5-20 to above 50 and hits 100 causing the high loads and when I need to enable the setting in Cloudflare until it drops to acceptable loads. I can see the status in Cloudflare showing the number of requests on the rise as well when this happens and php-fpm status page shows maxed out active processes. If I increase max_children, the load just increases dramatically as active processes reach their max_children. We are a local newsroom, the high traffic usually traces back to a news story getting shared or picked up by a national company. We got some recent national press, I had to leave the Cloudflare protection on most of the day and week following, and it has been coming in waves since.
So, I am trying to determine if it is time to add another server and look at load balancing or if some optimization is what’s needed. Or both. We are running Apache 2.4.62, PHP-FPM 8.3.24 and MariaDB 10.11.14.
Looking at top it seems php-fpm may be the cause of the high load at 98% CPU for one process during periods of high loads and the rest of the php-fpm processes 10-40% with the MariaDB processlist showing little with no queries more than 0 time, I may have one pop in there for 2-3 seconds, but nothing more and I never see more than 20-25 sleep processes during the high load.
As far as php-fpm, here are the current pertinent settings with typical system info during high loads…
pm = ondemand
pm.max_children = 15
pm.process_idle_timeout = 10s;
pm.max_requests = 500
ATOP - alma1 2025/08/26 18:37:13 ----------------- 10s elapsed
PRC | sys 4.37s | user 49.41s | #proc 189 | #zombie 0 | #exit 10 |
CPU | sys 41% | user 494% | irq 4% | idle 61% | wait 1% |
cpu | sys 7% | user 86% | irq 1% | idle 6% | cpu003 w 0% |
cpu | sys 6% | user 84% | irq 1% | idle 9% | cpu004 w 0% |
cpu | sys 7% | user 83% | irq 1% | idle 9% | cpu001 w 0% |
cpu | sys 7% | user 80% | irq 1% | idle 12% | cpu005 w 0% |
cpu | sys 7% | user 80% | irq 1% | idle 12% | cpu002 w 0% |
cpu | sys 7% | user 79% | irq 1% | idle 13% | cpu000 w 0% |
CPL | avg1 9.03 | avg5 9.64 | avg15 8.23 | csw 81558 | intr 104638 |
MEM | tot 15.4G | free 250.5M | buff 91.9M | slab 624.6M | numnode 1 |
SWP | tot 512.0M | free 8.3M | swcac 5.9M | vmcom 16.1G | vmlim 8.2G |
PAG | numamig 0 | migrate 14e3 | swin 0 | swout 0 | oomkill 0 |
DSK | sda | busy 1% | read 151 | write 517 | avio 0.19 ms |
NET | transport | tcpi 13312 | tcpo 26862 | udpi 891 | udpo 891 |
NET | network | ipi 14203 | ipo 19189 | ipfrw 0 | deliv 14203 |
NET | eth0 ---- | pcki 14197 | pcko 19173 | si 3192 Kbps | so 28 Mbps |
NET | lo ---- | pcki 16 | pcko 16 | si 4 Kbps | so 4 Kbps |
PID SYSCPU USRCPU RDELAY VGROW RGROW RDDSK WRDSK S CPUNR CPU CMD 1/5
2064409 0.16s 8.07s 1.52s 112.8M 45.7M 5.1M 0B R 1 82% php-fpm
2036955 0.25s 3.31s 1.27s -4.0M -4.2M 0B 0B S 0 36% php-fpm
2035984 0.29s 3.26s 0.81s 6.0M 133.0M 0B 0B R 4 36% php-fpm
2032153 0.24s 3.16s 1.44s 2.0M 2.0M 0B 0B S 5 34% php-fpm
2032982 0.22s 3.06s 1.36s 8.0M 8.1M 0B 0B S 1 33% php-fpm
2036323 0.22s 2.95s 1.50s 0B 112.0K 0B 0B S 2 32% php-fpm
2036736 0.22s 2.86s 1.28s 2.0M 2.2M 0B 0B S 5 31% php-fpm
2063710 0.28s 2.62s 1.66s 0B 129.3M 0B 0B R 0 29% php-fpm
2039311 0.22s 2.67s 1.12s 0B 0B 0B 0B R 0 29% php-fpm
2035483 0.22s 2.65s 0.95s 0B 0B 0B 0B R 2 29% php-fpm
2033448 0.24s 2.57s 1.22s 4.0M 3.4M 0B 0B S 4 28% php-fpm
2032951 0.21s 2.57s 1.45s -2.0M -1.1M 0B 0B S 0 28% php-fpm
2031692 0.21s 2.41s 0.97s -40.2M -34.5M 0B 0B S 0 26% php-fpm
2032826 0.21s 2.38s 1.24s 2.0M 2.0M 0B 0B R 5 26% php-fpm
3480718 0.60s 1.63s 5m46s 0B 13.6M 16.0K 2.3M S 0 22% mariadbd
2064719 0.17s 1.49s 0.43s 841.9M 210.9M 0B 0B S 2 17% php-fpm
Server load: 9.38 9.71 8.25
Total accesses: 16713 - Total Traffic: 1.4 GB - Total Duration: 22404839
CPU Usage: u7.5 s2.7 cu15.16 cs5.47 - 1.77% CPU load
9.62 requests/sec - 0.8 MB/second - 90.0 kB/request - 1340.56 ms/request
29 requests currently being processed, 0 workers gracefully restarting, 67 idle workers
Seeing what is taking my memory, I have this script…
[root@alma1 ~]# cat apache_mem_usage.sh
#!/usr/bin/sh
ps -ylC httpd | awk '{x += $8;y += 1} END {print "Apache memory usage (MB): "x/1024; print "Average Apache Process Size (MB): "x/((y-1)*1024)}'
ps -ylC php-fpm | awk '{x += $8;y += 1} END {print "PHP-FPM memory usage (MB): "x/1024; print "Average PHP-FPM Process Size (MB): "x/((y-1)*1024)}'
ps -ylC mariadbd | awk '{x += $8;y += 1} END {print "MariaDB memory usage (MB): "x/1024}'
free
[root@alma1 ~]# ./apache_mem_usage.sh
Apache memory usage (MB): 279.188
Average Apache Process Size (MB): 27.9187
PHP-FPM memory usage (MB): 5103.15
Average PHP-FPM Process Size (MB): 318.947
MariaDB memory usage (MB): 6707.73
total used free shared buff/cache available
Mem: 16113200 9426012 307164 441444 7163324 6687188
Swap: 524284 516656 7628
Perhaps the PHP-FPM process memory usage is higher than it should be? The innodb_buffer_pool_size parameter for MariaDB is set to 6G and I found the query that shows it could possibly need to be 3 times that size or some db optimization may be needed…
MariaDB [(none)]> SELECT CEILING(Total_InnoDB_Bytes*1.6/POWER(1024,3)) RIBPS FROM (SELECT SUM(data_length+index_length) Total_InnoDB_Bytes FROM information_schema.tables WHERE engine='InnoDB') A;
+-------+
| RIBPS |
+-------+
| 18 |
+-------+
So I don’t believe I can give up any of that memory for PHP unless I reduce that number. The largest innodb table I have is 19 million rows that tracks ads and has been running for many years with 3 indexes…
MariaDB [wmnf_www]> SHOW TABLE STATUS LIKE 'wp_ad_tracker' G;
*************************** 1. row ***************************
Name: wp_ad_tracker
Engine: InnoDB
Version: 10
Row_format: Dynamic
Rows: 19829074
Avg_row_length: 52
Data_length: 1047511040
Max_data_length: 0
Index_length: 1000275968
Data_free: 4194304
Auto_increment: 146764111
Create_time: 2025-04-20 09:53:51
Update_time: 2025-08-26 17:39:10
Check_time: NULL
Collation: utf8mb4_unicode_ci
Checksum: NULL
Create_options:
Comment:
Max_index_length: 0
Temporary: N
I’ll leave it at that for now, looking for pointers or recommendations on best way to proceed to get the loads reduced.