[olug] Need assistance from apache gurus

Mon Sep 19 20:11:01 CDT 2016

Jay -

I upped the Nofile and nproc last week to 10240 for each.

I just turned off iptables, and it seems faster - it hasn't hesitated on me
yet.  I've compared the two iptables config files, and they're pretty much
the same, so it's possible its something in the netfilter settings.  The
centos 5 system didn't have an entry for /proc/sys/net/nf_conntrack_max but
on centos 6 it was 65536.

The engineers at the colo suggested I change the VMware hardware settings
so the NIC was vmxnet3 instead of e1000.

I went through sysctl.conf and duplicated the settings over from centos 5
shortly after you sent your reply.

sysctl

net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_synack_retries = 2
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv6.conf.default.router_solicitations = 0
net.ipv6.conf.default.accept_ra_rtr_pref = 0
net.ipv6.conf.default.accept_ra_pinfo = 0
net.ipv6.conf.default.accept_ra_defrtr = 0
net.ipv6.conf.default.autoconf = 0
net.ipv6.conf.default.dad_transmits = 0
net.ipv6.conf.default.max_addresses = 1
kernel.exec-shield = 1
kernel.randomize_va_space = 1
fs.file-max = 65535
kernel.pid_max = 65536
net.ipv4.ip_local_port_range = 2000 65000
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296

This will be behind another firewall and load balancer, so I may be able to
skip by without iptables, but I hate that idea.  I've always had firewalls
on my systems, no matter the environment.

Quick ab test (8 GB RAM, 4 VCPUs)

Server Software:        Apache/2.2.15
Server Hostname:        xxxxxxxxxxxxxxxxxxxxxxx
Server Port:            80

Document Path:          /
Document Length:        56158 bytes

Concurrency Level:      5
Time taken for tests:   27.745 seconds
Complete requests:      10
Failed requests:        9
   (Connect: 0, Receive: 0, Length: 9, Exceptions: 0)
Total transferred:      566215 bytes
HTML transferred:       562375 bytes
Requests per second:    0.36 [#/sec] (mean)
Time per request:       13872.730 [ms] (mean)
Time per request:       2774.546 [ms] (mean, across all concurrent requests)
Transfer rate:          19.93 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       60   63   4.3     60      69
Processing:  4193 13697 9880.6  22881   23291
Waiting:     3713 13189 9643.0  21051   22720
Total:       4256 13759 9877.3  22940   23351

Percentage of the requests served within a certain time (ms)
  50%  22940
  66%  23059
  75%  23067
  80%  23227
  90%  23351
  95%  23351
  98%  23351
  99%  23351
 100%  23351 (longest request)

(home page is dynamic content, 2.5 MB in size.)

I'll be doing some more testing when I get back in the office in the
morning.

Jon L.

On Mon, Sep 19, 2016 at 4:48 PM, Jay Bendon <jaybocc2 at gmail.com> wrote:

> How do TCP settings compare from centos5 to centos6?
>
> Are you getting a lot of syn drops? (netstat -s |grep -i dropped)
>
> NoFiles limits reasonable?
>
> sanity check your somaxconn and tcp_max_syn_backlog and rmem wmem, and
> nf_conntrack_max (if using iptables) settings for your application (and
> compare to centos5)
>
> Just spitballing some of the common reasons for connectivity issues under
> loads
>
> --Jay
>
> On Mon, Sep 19, 2016 at 2:15 PM, Jon Larsen <jon at jonlarsen.us> wrote:
>
> > I have a weird issue on my plate.
> >
> > I have three apache web servers behind an ldirector load balancer running
> > centos 5.x on VMWARE.  I've built three new centos 6.x web server VMs to
> > replace them.
> >
> > I used the same apache configs, as the apache versions don't change much
> > between 5 and 6.
> >
> > I'm encountering intermittent network disconnects when I use the new
> three
> > centos 6 systems in production, forcing me to back peddle to the older
> cent
> > 5 systems.
> >
> > The disconnects appear at random, and no concurrent high CPU load.
> >
> > The disk scheduler is already set for deadline, and I'm using the
> suggested
> > VMware 'vmxnet3' nic adapters.
> >
> > I've tried several profiles of prefork settings, but encounter the same
> > issue.
> >
> > Currently, they are set to:
> >
> > StartServers 100
> > MinSpareServers 30
> > MaxSpareServers 40
> > ServerLimit 220
> > MaxClients 220
> > MaxRequestsperChild 2000
> >
> > Any ideas?
> >
> > Jon L.
> > _______________________________________________
> > OLUG mailing list
> > OLUG at olug.org
> > https://lists.olug.org/mailman/listinfo/olug
> >
> _______________________________________________
> OLUG mailing list
> OLUG at olug.org
> https://lists.olug.org/mailman/listinfo/olug
>