Strange performance with CPU with Realm Architecture

[NoobShinken]

hi Akoo,

All my process run on the same VM : arbitrer, Broker, scheduler, receiver and poller.
i've installed an distributed architecture with realm.

I try to change my config by replacing localhost on receiver.cfg per 127.0.0.1 or his Ip adress but it's doesn't change anything.

It's seems to waite something or waiting a config...

on /var/log/shinken/receiver.log

[1409212384] DEBUG: [receiver-master] Debug perf: get_external_commands [args:4.05311584473e-06] [aqu_lock:0.0] [calling:2.09808349609e-05] [json:1.09672546387e-05] [global:3.60012054443e-05]
[1409212385] DEBUG: [receiver-master] Debug perf: got_conf [args:5.00679016113e-06] [aqu_lock:9.53674316406e-07] [calling:2.86102294922e-06] [json:2.31266021729e-05] [global:3.19480895996e-05]
[1409212385] DEBUG: [receiver-master] Debug perf: ping [args:4.05311584473e-06] [aqu_lock:0.0] [calling:9.53674316406e-07] [json:1.09672546387e-05] [global:1.59740447998e-05]
[1409212385] DEBUG: [receiver-master] Debug perf: get_external_commands [args:9.05990600586e-06] [aqu_lock:9.53674316406e-07] [calling:8.48770141602e-05] [json:7.15255737305e-06] [global:0.000102043151855]
[1409212386] DEBUG: [receiver-master] Debug perf: got_conf [args:5.00679016113e-06] [aqu_lock:0.0] [calling:4.05311584473e-06] [json:2.28881835938e-05] [global:3.19480895996e-05]
[1409212386] DEBUG: [receiver-master] Debug perf: ping [args:4.05311584473e-06] [aqu_lock:9.53674316406e-07] [calling:1.19209289551e-06] [json:5.00679016113e-06] [global:1.12056732178e-05]
[1409212386] DEBUG: [receiver-master] Debug perf: get_external_commands [args:3.81469726562e-06] [aqu_lock:5.96046447754e-06] [calling:1.62124633789e-05] [json:5.96046447754e-06] [global:3.19480895996e-05]

from /var/log/shinken/arbitrer.log

[1409212103] INFO: [Shinken] [All] Dispatching receiver satellite with order: receiver-master (spare:False),
[1409212103] INFO: [Shinken] [All] Trying to send configuration to receiver receiver-master
[1409212103] INFO: [Shinken] [All] Dispatch OK of configuration 0 to receiver receiver-master
[1409212103] INFO: [Shinken] (http://10.38.231.189:7773/)
[1409212103] INFO: [Shinken] [All] OK, no more receiver sent need
[1409212103] INFO: [Shinken] [Test] Dispatching reactionner satellite with order:
[1409212103] INFO: [Shinken] [Test] OK, no more reactionner sent need
[1409212103] INFO: [Shinken] [Test] Dispatching poller satellite with order: poller-test (spare:False),
[1409212103] INFO: [Shinken] [Test] Trying to send configuration to poller poller-test
[1409212103] INFO: [Shinken] [Test] Dispatch OK of configuration 1 to poller poller-test
[1409212103] INFO: [Shinken] [Test] OK, no more poller sent need
[1409212103] INFO: [Shinken] [Test] Dispatching broker satellite with order: broker-master (spare:False),
[1409212103] INFO: [Shinken] [Test] Trying to send configuration to broker broker-master
[1409212103] INFO: [Shinken] [Test] Dispatch OK of configuration 1 to broker broker-master
[1409212103] INFO: [Shinken] [Test] OK, no more broker sent need
[1409212103] INFO: [Shinken] [Test] Dispatching receiver satellite with order: receiver-master (spare:False),
[1409212103] INFO: [Shinken] [Test] Trying to send configuration to receiver receiver-master
[1409212103] INFO: [Shinken] [Test] Dispatch OK of configuration 1 to receiver receiver-master
[1409212103] INFO: [Shinken] (http://10.38.231.189:7773/)
[1409212103] INFO: [Shinken] [Test] OK, no more receiver sent need
[1409212103] INFO: [Shinken] [All] Trying to send configuration to receiver receiver-master
[1409212103] INFO: [Shinken] [All] Dispatch OK of configuration to receiver receiver-master
[1409212104] TIMEPERIOD TRANSITION: us-holidays;-1;0
[1409212104] TIMEPERIOD TRANSITION: none;-1;0
[1409212104] TIMEPERIOD TRANSITION: 24x7;-1;1
[1409212104] TIMEPERIOD TRANSITION: workhours;-1;1

Configuration seems ok from this log.

[NoobShinken]

i forget this is my config on receiver

define receiver {
receiver_name receiver-master
#address localhost
address 10.38.231.189
port 7773
spare 0

## Optional parameters
timeout 3 ; Ping timeout
data_timeout 120 ; Data send timeout
max_check_attempts 3 ; If ping fails N or more, then the node is dead
check_interval 60 ; Ping node every N seconds

## Modules for Receiver
# - named-pipe = Open the named pipe nagios.cmd
# - nsca = NSCA server
# - tsca = TSCA server
# - ws-arbiter = WebService for pushing results to the arbiter
# - collectd = Receive collectd perfdata
modules

# Enable https or not
use_ssl 0
# enable certificate/hostname check, will avoid man in the middle attacks
hard_ssl_name_check 0

## Advanced Feature
direct_routing 0 ; If enabled, it will directly send commands to the
; schedulers if it knows about the hostname in the
; command.
realm All
}

[Akoo]

Hmm,
Configuration is good, nothing relevant on the log. I don't know why only receiver daemon will use a lot of CPU like that.
Try your configuration an another server if it's possible.
Did you have other VMs ? Which kind of hypervisor did you have ?

On my side, I have shinken 2.0.3 on debian 7.5. My master server is an VM too with only arbiter, receiver and reactionner. He is very calm for a lot of checks running.

Cheers

[NoobShinken]

Hi,

I just have one central on a VM with all modules and another VM on for realm wich have one scheduler and poller
All The VM are managed on my only one ESX because i'm testing SHinken distributed architecture.

I can show you my config via TeamViewer if you want by MP.

A strange behaviour with restart shinken service:

on my central, if i do /etc/init.d/shinken stop and after start----> CPU overload
and if i do /etc/init.d/shinken restart -----> CPU is OK

Very strange no?

[naparuba]

an you strace it? It's a very very strange behavior indeed

[NoobShinken]

hi,

i've installed the HA configuration and i have too on my spare server, an overload CPU for module receiver

i've had to stop this module to have a normal CPU

PS: now /etc/init.d/shinken stop and after start and restart shinken doesn't change anything ---> CPU is overload