View Issue Details

IDProjectCategoryView StatusLast Update
0003829Rocky-Linux-9mod_proxy_clusterpublic2023-08-30 11:40
ReporterMarco Benuzzi Assigned ToLouis Abel  
PrioritynormalSeveritymajorReproducibilityalways
Status needinfoResolutionopen 
Summary0003829: shipped version don't works
DescriptionI'm preparing a Rocky 9 server to substitute an old CentOS 7.
This server has an httpd with mod_cluster to balance backend JBoss machines.
I've installed using dnf:
- httpd Version: 2.4.53 Release: 11.el9_2.5
- mod_proxy_cluster Version: 1.3.18 Release: 1.el9

The configuration of httpd is more or less the same of the old CentOS 7 machine.
The mod_cluster status page shows all the JBoss nodes.

But any HTTP request receives HTTP 503 Service Unavailable.
The message in the HTTP error log is
proxy: CLUSTER: (balancer://cluster). All workers are in error state

After a long debugging without success, I finally downloaded the mod_proxy_cluster sources from https://github.com/modcluster/mod_proxy_cluster.git and compiled enabling extended debug message to dig what is happining.
But, using the compiled version of mod_proxy_cluster, all is working as expected.

It seems that the mod_proxy_cluster shipped is broken.
Additional Informationoriginal configuration
LoadModule proxy_cluster_module /usr/lib64/httpd/modules/mod_proxy_cluster.so
LoadModule cluster_slotmem_module /usr/lib64/httpd/modules/mod_cluster_slotmem.so
LoadModule manager_module /usr/lib64/httpd/modules/mod_manager.so
LoadModule advertise_module /usr/lib64/httpd/modules/mod_advertise.so

modified configuration
LoadModule proxy_cluster_module modules.modcluster/mod_proxy_cluster.so
LoadModule manager_module modules.modcluster/mod_manager.so
LoadModule advertise_module modules.modcluster/mod_advertise.so
LoadModule lbmethod_cluster_module modules.modcluster/mod_lbmethod_cluster.so

Note that in the last version of mod_proxy_cluster mod_cluster_slotmem is substitued by mod_lbmethod_cluster
TagsNo tags attached.

Activities

Louis Abel

Louis Abel

2023-07-20 23:33

administrator   ~0004126

We do not have enough reproduction steps. Please provide the configuration you are using with httpd so we can effectively reproduce your issue.

Setting to needinfo.
Marco Benuzzi

Marco Benuzzi

2023-08-29 15:49

reporter   ~0004456

To repoduce the problem I've created a VM with 3 network cards and installed Rocky Linux 9.2 minimal.

My IP addresses are:
- IP1 192.168.122.80 used for apache
- IP2 192.168.122.186 used for jboss node 1
- IP3 192.168.122.235 used for jboss node 2

After first boot:
- dnf update
- dnf install httpd mod_proxy_cluster java-1.8.0-openjdk-devel
- in /etc/sysconfig/selinux change SELINUX=permissive
- firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="IP1/24" accept'
- reboot

Configure httpd
- put httpd.conf in /etc/httpd/conf
- in /etc/httpd/conf/httpd.conf change lines 2 and 3 (Listen) and lines 75 and 93 (VirtualHost) with your IP1 [don't change ports]
- mv /etc/httpd/mod_proxy_cluster.conf.sample /etc/httpd/conf.modules.d/10-cluster.conf
- in /etc/httpd/conf.modules.d/00-proxy.conf comment line 8 #LoadModule proxy_balancer_module modules/mod_proxy_balancer.so

Download JBoss EAP 6.3 and unzip in two directories (node1 and node2)

Configure JBoss node1
- put standalone.conf in ~/node1/bin and change line 50 THISIP=192.168.122.186 with your IP2
- put standalone.xml in ~/node1/standalone/configuration and change line 433 proxy-list="192.168.122.80:81" with your IP1 [don't change port]
- put jboss-helloworld.war in ~/node1/standalone/deployments/
- touch ~/node1/standalone/deployments/jboss-helloworld.war.dodeploy

Configure JBoss node2
- put standalone.conf in ~/node2/bin and change line 50 THISIP=192.168.122.186 with your IP3
- put standalone.xml in ~/node2/standalone/configuration and change line 433 proxy-list="192.168.122.80:81" with your IP1 [don't change port]
- put jboss-helloworld.war in ~/node2/standalone/deployments/
- touch ~/node2/standalone/deployments/jboss-helloworld.war.dodeploy

Start services:
- systemctl start httpd
- ~/node1/bin/standalone.sh
- ~/node2/bin/standalone.sh

Checks:
- apache is running http://IP1/server-status
- cluster is running http://IP1/cluster-status (see cluster-status.png)
- application is running on node1 http://IP2:8080/jboss-helloworld
- application is running on node2 http://IP3:8080/jboss-helloworld
- no error messages in /var/log/httpd/cluster_error_log

Test application on cluster http://IP1/jboss-helloworld
HTTP status 503 service unavailable
For each request a error i written in /var/log/httpd/error_log
[Tue Aug 29 17:47:41.978206 2023] [:error] [pid 17034:tid 17077] proxy: CLUSTER: (balancer://cluster). All workers are in error state
cluster-status.png (99,555 bytes)   
cluster-status.png (99,555 bytes)   
comment1.txt (2,451 bytes)   
To repoduce the problem I've created a VM with 3 network cards and installed Rocky Linux 9.2 minimal.

My IP addresses are:
- IP1 192.168.122.80 used for apache
- IP2 192.168.122.186 used for jboss node 1
- IP3 192.168.122.235 used for jboss node 2

After first boot:
- dnf update
- dnf install httpd mod_proxy_cluster java-1.8.0-openjdk-devel
- in /etc/sysconfig/selinux change SELINUX=permissive
- firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="IP1/24" accept'
- reboot

Configure httpd
- put httpd.conf in /etc/httpd/conf
- in /etc/httpd/conf/httpd.conf change lines 2 and 3 (Listen) and lines 75 and 93 (VirtualHost) with your IP1 [don't change ports]
- mv /etc/httpd/mod_proxy_cluster.conf.sample /etc/httpd/conf.modules.d/10-cluster.conf
- in /etc/httpd/conf.modules.d/00-proxy.conf comment line 8 #LoadModule proxy_balancer_module modules/mod_proxy_balancer.so

Download JBoss EAP 6.3 and unzip in two directories (node1 and node2)

Configure JBoss node1
- put standalone.conf in ~/node1/bin and change line 50 THISIP=192.168.122.186 with your IP2
- put standalone.xml in ~/node1/standalone/configuration and change line 433 proxy-list="192.168.122.80:81" with your IP1 [don't change port]
- put jboss-helloworld.war in ~/node1/standalone/deployments/
- touch ~/node1/standalone/deployments/jboss-helloworld.war.dodeploy

Configure JBoss node2
- put standalone.conf in ~/node2/bin and change line 50 THISIP=192.168.122.186 with your IP3
- put standalone.xml in ~/node2/standalone/configuration and change line 433 proxy-list="192.168.122.80:81" with your IP1 [don't change port]
- put jboss-helloworld.war in ~/node2/standalone/deployments/
- touch ~/node2/standalone/deployments/jboss-helloworld.war.dodeploy

Start services:
- systemctl start httpd
- ~/node1/bin/standalone.sh
- ~/node2/bin/standalone.sh

Checks:
- apache is running http://IP1/server-status
- cluster is running http://IP1/cluster-status (see cluster-status.png)
- application is running on node1 http://IP2:8080/jboss-helloworld
- application is running on node2 http://IP3:8080/jboss-helloworld
- no error messages in /var/log/httpd/cluster_error_log

Test application on cluster http://IP1/jboss-helloworld
HTTP status 503 service unavailable
For each request a error i written in /var/log/httpd/error_log
[Tue Aug 29 17:47:41.978206 2023] [:error] [pid 17034:tid 17077] proxy: CLUSTER: (balancer://cluster). All workers are in error state

comment1.txt (2,451 bytes)   
Marco Benuzzi

Marco Benuzzi

2023-08-29 15:51

reporter   ~0004457

Missed attachments
Marco Benuzzi

Marco Benuzzi

2023-08-29 15:52

reporter   ~0004458

Missed attachments (zipped)
Marco Benuzzi

Marco Benuzzi

2023-08-29 15:54

reporter   ~0004459

I got error submitting with attachment...
Marco Benuzzi

Marco Benuzzi

2023-08-29 15:55

reporter   ~0004460

Trying with tar.gz
attachments.tar.gz (13,555 bytes)
Marco Benuzzi

Marco Benuzzi

2023-08-30 11:38

reporter   ~0004465

I've tested with mod_proxy_cluster compiled from sources.

With version 1.3.x there is always the same problem.

The only version that works is the 2.x
Marco Benuzzi

Marco Benuzzi

2023-08-30 11:40

reporter   ~0004466

Note that in Centos 7 with httpd 2.4.6 and mod_proxy_cluster 1.3.1 all is working well.

Issue History

Date Modified Username Field Change
2023-07-20 17:08 Marco Benuzzi New Issue
2023-07-20 23:33 Louis Abel Assigned To => Louis Abel
2023-07-20 23:33 Louis Abel Status new => needinfo
2023-07-20 23:33 Louis Abel Note Added: 0004126
2023-08-29 15:49 Marco Benuzzi Note Added: 0004456
2023-08-29 15:49 Marco Benuzzi File Added: cluster-status.png
2023-08-29 15:49 Marco Benuzzi File Added: comment1.txt
2023-08-29 15:51 Marco Benuzzi Note Added: 0004457
2023-08-29 15:52 Marco Benuzzi Note Added: 0004458
2023-08-29 15:54 Marco Benuzzi Note Added: 0004459
2023-08-29 15:55 Marco Benuzzi Note Added: 0004460
2023-08-29 15:55 Marco Benuzzi File Added: attachments.tar.gz
2023-08-30 11:38 Marco Benuzzi Note Added: 0004465
2023-08-30 11:40 Marco Benuzzi Note Added: 0004466