Page 1 of 2 12 LastLast
Results 1 to 10 of 16

Thread: Strange performance with CPU with Realm Architecture

  1. #1
    Junior Member
    Join Date
    Aug 2014
    Posts
    19

    Strange performance with CPU with Realm Architecture

    Hi all,

    It's again me ;D,

    I have a distributed architecture with realm config.
    I have a strange performance with the CPU when the realm is activate.
    My CPU of my central serveur is between 45-50% for just check 3 hosts (2 realm All and 1 another realm)
    But if i disable the realm ( a normal config with no realm defined) and i check the same hosts (all in the central) my cpu is normal (~0-1%).

    I don't know why??

    I noticed that when i used an archi distributed with realm, the cpu will come normal if i stop the service shinken-receiver .
    I dont know why this service take a lot of CPU???

    /var/log/shinken/receiverd.log:
    [1408541176] WARNING: [Shinken] Printing stored debug messages prior to our daemonization
    [1408541176] INFO: [Shinken] Modules directory: /var/lib/shinken/modules
    [1408541176] INFO: [Shinken] Waiting for initial configuration
    [1408541176] INFO: [Shinken] Starting HTTP daemon
    [1408541177] INFO: [receiver-master] We received modules []
    [1408541177] INFO: [receiver-master] I correctly loaded the modules: []
    [1408541178] INFO: [receiver-master] [receiver-master] We already got the conf 0 (scheduler-master)
    [1408541178] INFO: [receiver-master] [receiver-master] We already got the conf 0 (scheduler-master)
    [1408541178] INFO: [receiver-master] [receiver-master] We already got the conf 1 (scheduler-test)
    [1408541178] INFO: [receiver-master] [receiver-master] We already got the conf 0 (scheduler-master)
    [1408541178] INFO: [receiver-master] [receiver-master] We already got the conf 1 (scheduler-test)
    [1408541268] INFO: [receiver-master] [receiver-master] Stopping all workers
    [1408541268] INFO: [receiver-master] [receiver-master] Stopping all network connections
    [1408541294] INFO: [Shinken] Using the local log file '/var/log/shinken/receiverd.log'
    [1408541294] WARNING: [Shinken] Printing stored debug messages prior to our daemonization
    [1408541294] INFO: [Shinken] Modules directory: /var/lib/shinken/modules
    [1408541294] INFO: [Shinken] Waiting for initial configuration
    [1408541294] INFO: [Shinken] Starting HTTP daemon
    [1408541296] INFO: [receiver-master] We received modules []
    [1408541296] INFO: [receiver-master] I correctly loaded the modules: []
    [1408542026] INFO: [receiver-master] [receiver-master] Stopping all workers
    [1408542026] INFO: [receiver-master] [receiver-master] Stopping all network connections
    If someone can help me ?
    thank ;D


  2. #2
    Junior Member
    Join Date
    Aug 2014
    Posts
    19

    Re: Strange performance with CPU with Realm Architecture

    i've found the problem but don't know how to resolve it

    my cpu is overload when i install the package for oracle-instant-client
    oracle-instantclient-basic-10.2.0.3-1.i386.rpm
    oracle-instantclient-devel-10.2.0.3-1.i386.rpm
    oracle-instantclient-sqlplus-10.2.0.3-1.i386.rpm

    i need these package to use sqlplus for connect to remote database oracle to do some check or very some data....


  3. #3
    Shinken project leader
    Join Date
    May 2011
    Location
    Bordeaux (France)
    Posts
    2,130

    Re: Strange performance with CPU with Realm Architecture

    Important question: which process is usng the cpu?
    No direct support by personal message. Please open a thread so everyone can see the solution

  4. #4
    Junior Member
    Join Date
    Aug 2014
    Posts
    19

    Re: Strange performance with CPU with Realm Architecture

    it's the process Pyhton which take a lot of cpu

    So to avoid the problem for the moment, i've installed these package oracle-instant-client on the satellite and i've desintalled them from central.
    So now all my check for oracle will be done by the satellite and not from central

  5. #5
    Shinken project leader
    Join Date
    May 2011
    Location
    Bordeaux (France)
    Posts
    2,130

    Re: Strange performance with CPU with Realm Architecture

    I don't see the impact of theses packages on python, it do not use them at all. If it was the perl oracle checks yes, it can explain this, but not python. which python process (broker, scheduler, poller,..)?
    No direct support by personal message. Please open a thread so everyone can see the solution

  6. #6
    Junior Member
    Join Date
    Aug 2014
    Posts
    19

    Re: Strange performance with CPU with Realm Architecture

    Hi,

    The process which takes a lot of cpu it's python process for receiver.
    when i stopped this services, my cpu is normal (0-1% cpu) and the check is done great.
    But when i started this service,my cpu is between 45-50% and the check is done great too.

    For my oracle check,i use my perl script because i had to check some specific data and i used them on my nagios monitoring.

    Thank for your help narupa

  7. #7
    Junior Member
    Join Date
    Aug 2014
    Posts
    19

    Re: Strange performance with CPU with Realm Architecture

    A very strange Performance of CPU, i 've desactivated all config and just let one host configured and i use pack linux-snmp,ssh and http.
    I've added on my central shinken 2 CPU + 2048Mo of RAM too.

    ---> The cpu is overload, i don't know why???


    /etc/shinken/hosts/localhost.cfg:
    define host{
    #use generic-host
    #use generic-host,ssh,linux-snmp
    use generic-host,ssh,linux-snmp,http
    # use generic-linux-host,http
    contact_groups admins
    host_name localhost
    address localhost
    }
    and the process shinken-receiver take a lot of cpu ???

    extract of ps
    root@shinken-central:~# ps -aux --forest| grep shinken
    warning: bad ps syntax, perhaps a bogus '-'?
    See http://gitorious.org/procps/procps/b...umentation/FAQ
    shinken 2103 0.0 0.0 10488 908 ? S 15:04 0:00 /usr/local/pnp4nagios/bin/npcd -d -f /usr/local/pnp4nagios/etc/npcd.cfg
    root 1499 0.0 0.0 3552 792 pts/1 S+ 15:26 0:00 \_ grep shinken
    shinken 981 0.3 0.3 108964 15708 ? Sl 15:18 0:01 python /usr/bin/shinken-scheduler -d -c /etc/shinken/daemons/schedulerd.ini
    shinken 983 0.0 0.2 21412 9620 ? S 15:18 0:00 \_ python /usr/bin/shinken-scheduler -d -c /etc/shinken/daemons/schedulerd.ini
    shinken 1024 0.2 0.3 109508 15900 ? Sl 15:18 0:01 python /usr/bin/shinken-poller -d -c /etc/shinken/daemons/pollerd.ini
    shinken 1026 0.1 0.2 103160 8848 ? Sl 15:18 0:00 \_ python /usr/bin/shinken-poller -d -c /etc/shinken/daemons/pollerd.ini
    shinken 1250 0.0 0.3 109116 13824 ? S 15:18 0:00 \_ python /usr/bin/shinken-poller -d -c /etc/shinken/daemons/pollerd.ini
    shinken 1264 0.0 0.3 109128 13340 ? S 15:18 0:00 \_ python /usr/bin/shinken-poller -d -c /etc/shinken/daemons/pollerd.ini
    shinken 1275 0.0 0.3 109140 13840 ? S 15:18 0:00 \_ python /usr/bin/shinken-poller -d -c /etc/shinken/daemons/pollerd.ini
    shinken 1290 0.0 0.3 109152 13356 ? S 15:18 0:00 \_ python /usr/bin/shinken-poller -d -c /etc/shinken/daemons/pollerd.ini
    shinken 1314 0.0 0.3 109164 13864 ? S 15:18 0:00 \_ python /usr/bin/shinken-poller -d -c /etc/shinken/daemons/pollerd.ini
    shinken 1330 0.0 0.3 109176 13424 ? S 15:18 0:00 \_ python /usr/bin/shinken-poller -d -c /etc/shinken/daemons/pollerd.ini
    shinken 1369 0.0 0.3 109188 13848 ? S 15:18 0:00 \_ python /usr/bin/shinken-poller -d -c /etc/shinken/daemons/pollerd.ini
    shinken 1376 0.0 0.3 109200 13428 ? S 15:18 0:00 \_ python /usr/bin/shinken-poller -d -c /etc/shinken/daemons/pollerd.ini
    shinken 1070 0.1 0.3 108376 15684 ? Sl 15:18 0:00 python /usr/bin/shinken-reactionner -d -c /etc/shinken/daemons/reactionnerd.ini
    shinken 1072 0.0 0.2 36568 8456 ? Sl 15:18 0:00 \_ python /usr/bin/shinken-reactionner -d -c /etc/shinken/daemons/reactionnerd.ini
    shinken 1359 0.0 0.3 108104 13512 ? S 15:18 0:00 \_ python /usr/bin/shinken-reactionner -d -c /etc/shinken/daemons/reactionnerd.ini
    shinken 1116 0.2 0.3 109852 16064 ? Sl 15:18 0:01 python /usr/bin/shinken-broker -d -c /etc/shinken/daemons/brokerd.ini
    shinken 1118 0.6 0.2 55876 10176 ? Sl 15:18 0:03 \_ python /usr/bin/shinken-broker -d -c /etc/shinken/daemons/brokerd.ini
    shinken 1278 0.0 0.4 119664 16896 ? Sl 15:18 0:00 \_ python /usr/bin/shinken-broker -d -c /etc/shinken/daemons/brokerd.ini
    shinken 1317 0.3 0.3 109928 15304 ? S 15:18 0:01 \_ python /usr/bin/shinken-broker -d -c /etc/shinken/daemons/brokerd.ini
    shinken 1343 0.0 0.3 109092 14172 ? S 15:18 0:00 \_ python /usr/bin/shinken-broker -d -c /etc/shinken/daemons/brokerd.ini
    shinken 1159 99.5 0.3 109364 15120 ? Rl 15:18 8:05 python /usr/bin/shinken-receiver -d -c /etc/shinken/daemons/receiverd.ini
    shinken 1161 0.0 0.2 21428 9348 ? S 15:18 0:00 \_ python /usr/bin/shinken-receiver -d -c /etc/shinken/daemons/receiverd.ini
    shinken 1202 0.2 0.3 110864 15696 ? Sl 15:18 0:01 python /usr/bin/shinken-arbiter -d -c /etc/shinken/shinken.cfg
    shinken 1204 0.0 0.3 27600 14144 ? S 15:18 0:00 \_ python /usr/bin/shinken-arbiter -d -c /etc/shinken/shinken.cfg
    extract of top
    top - 15:28:40 up 24 min, 2 users, load average: 1.02, 0.92, 0.58
    Tasks: 116 total, 2 running, 114 sleeping, 0 stopped, 0 zombie
    %Cpu(s): 23.3 us, 2.2 sy, 0.0 ni, 74.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
    KiB Mem: 4146916 total, 975184 used, 3171732 free, 37360 buffers
    KiB Swap: 552956 total, 0 used, 552956 free, 644976 cached

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    1159 shinken 20 0 106m 14m 2972 R 99.8 0.4 10:04.89 python
    1118 shinken 20 0 55876 9.9m 1384 S 0.7 0.2 0:04.07 python
    981 shinken 20 0 106m 15m 2956 S 0.3 0.4 0:01.93 python
    1024 shinken 20 0 106m 15m 3460 S 0.3 0.4 0:01.27 python
    1026 shinken 20 0 100m 8848 1436 S 0.3 0.2 0:00.71 python
    1116 shinken 20 0 107m 15m 3344 S 0.3 0.4 0:01.64 python
    1202 shinken 20 0 108m 15m 2132 S 0.3 0.4 0:01.28 python
    1317 shinken 20 0 107m 14m 2368 S 0.3 0.4 0:02.36 python
    2094 mongodb 20 0 193m 60m 39m S 0.3 1.5 0:03.37 mongod

  8. #8
    Junior Member
    Join Date
    Aug 2014
    Posts
    9

    Re: Strange performance with CPU with Realm Architecture

    I suggest you to enable DEBUG and also, you can use the command "strace"

    For example here:
    shinken 1159 99.5 0.3 109364 15120 ? Rl 15:18 8:05 python /usr/bin/shinken-receiver -d -c /etc/shinken/daemons/receiverd.ini
    shinken 1161 0.0 0.2 21428 9348 ? S 15:18 0:00 \_ python /usr/bin/shinken-receiver -d -c /etc/shinken/daemons/receiverd.ini
    Your process id is "1159"

    Try :
    strace -p 1159

    And forward output on a file.
    I think this should help to see the error.

    Also, can you please tell me which OS did you use and also which version ?

  9. #9
    Junior Member
    Join Date
    Aug 2014
    Posts
    19

    Re: Strange performance with CPU with Realm Architecture

    Hi Akoo

    i used a debian 7.6 which have 4 GO Ram and 4 CPu on a virtual Machine.

    i never touch about file config of shinken-receiver it's the config by default

    on my strace i got always this message :
    gettimeofday({1409152938, 822361}, NULL) = 0
    write(1, ".", 1) = 1
    waitpid(11866, 0xbf80792c, WNOHANG) = 0
    gettimeofday({1409152938, 822492}, NULL) = 0
    select(0, NULL, NULL, NULL, {0, 0}) = 0 (Timeout)
    gettimeofday({1409152938, 822561}, NULL) = 0
    gettimeofday({1409152938, 822585}, NULL) = 0
    write(1, ".", 1) = 1
    waitpid(11866, 0xbf80792c, WNOHANG) = 0
    gettimeofday({1409152938, 822673}, NULL) = 0
    select(0, NULL, NULL, NULL, {0, 0}) = 0 (Timeout)
    gettimeofday({1409152938, 822732}, NULL) = 0
    gettimeofday({1409152938, 822756}, NULL) = 0
    write(1, ".", 1) = 1
    waitpid(11866, 0xbf80792c, WNOHANG) = 0
    gettimeofday({1409152938, 822844}, NULL) = 0
    select(0, NULL, NULL, NULL, {0, 0}) = 0 (Timeout)
    gettimeofday({1409152938, 822903}, NULL) = 0
    gettimeofday({1409152938, 822926}, NULL) = 0
    write(1, ".", 1) = 1
    waitpid(11866, 0xbf80792c, WNOHANG) = 0
    gettimeofday({1409152938, 823013}, NULL) = 0
    select(0, NULL, NULL, NULL, {0, 0}) = 0 (Timeout)
    gettimeofday({1409152938, 823071}, NULL) = 0
    gettimeofday({1409152938, 823095}, NULL) = 0
    write(1, ".", 1) = 1
    waitpid(11866, 0xbf80792c, WNOHANG) = 0
    gettimeofday({1409152938, 823182}, NULL) = 0
    select(0, NULL, NULL, NULL, {0, 0}) = 0 (Timeout)
    gettimeofday({1409152938, 823240}, NULL) = 0
    gettimeofday({1409152938, 823264}, NULL) = 0
    write(1, ".", 1) = 1
    it's seems to wait a config ???

  10. #10
    Junior Member
    Join Date
    Aug 2014
    Posts
    9

    Re: Strange performance with CPU with Realm Architecture

    Hello,

    It's strange, can you confirm me that all your process run on the same VM ?
    The timeout message on the strace is not normal. Maybe he have some difficulty to resolv "localhost", check on your /etc/hosts or replace localhost per 127.0.0.1 on your configuration.


    Cheers

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •