Page 1 of 2 12 LastLast
Results 1 to 10 of 19

Thread: SnmpBooster, genDevConfig Pending STATE

  1. #1
    Junior Member
    Join Date
    May 2013
    Posts
    10

    SnmpBooster, genDevConfig Pending STATE

    Hello,

    This is my first Shinken installation (Ubuntu 12.04 server in vmware) I am running into problem when trying to monitor my Cisco switch with SnmpBooster.

    i followed both of theses procedures to install:
    http://www.shinken-monitoring.org/wi...en_10min_start
    http://www.shinken-monitoring.org/wi...booster_module

    I fixed the Ubuntu 12.04 Memcached errors using the SnmpBooster Troubleshooting document:
    http://www.shinken-monitoring.org/wi...roubleshooting

    I was able to generate the host config file with genDevConfig and everything look ok with SnmpBooster but in the WebUI all SNMP checks for the switch are in PENDING state.

    I turned on debugging, the only thing i can find in the logs is this:

    nagios.log
    Code:
    ...
    [1367609389] Info :  [Arbiter] Serializing the configurations...
    [1367609389] Info :  Configuration Loaded
    [1367609389] Info :  Trying to initialize additional groups for the daemon
    [1367609389] Warning : Stale pidfile exists (no or invalid or unreadable content). Reusing it.
    [1367609389] Info :  Initializing Pyro connection with host:localhost port:7770 ssl:False
    [1367609389] Warning : Printing stored debug messages prior to our daemonization
    [1367609389] Warning : IOError(2, 'No such file or directory')
    [1367609391] Warning : [poller-1] The worker 1 goes down unexpectedly!
    [1367609391] Info :  [poller-1] Allocating new fork Worker: 2
    [1367609391] Info :  [poller-1] Allocating new snmp_poller Worker: 3
    [1367609395] Warning : [poller-1] The worker 3 goes down unexpectedly!
    ...
    1367610772] Warning : 14 actions never came back for the satellite 'poller-1'. I'm reenable them for polling
    [1367610832] Warning : 14 actions never came back for the satellite 'poller-1'. I'm reenable them for polling
    [1367610893] Warning : 14 actions never came back for the satellite 'poller-1'. I'm reenable them for polling
    [1367610928] Warning : [reactionner-1] The worker 0 goes down unexpectedly!
    [1367610928] Info :  [reactionner-1] Allocating new fork Worker: 1
    [1367610953] Warning : 14 actions never came back for the satellite 'poller-1'. I'm reenable them for polling
    [1367611013] Warning : 14 actions never came back for the satellite 'poller-1'. I'm reenable them for polling
    [1367611073] Warning : 14 actions never came back for the satellite 'poller-1'. I'm reenable them for polling
    ...
    arbiter.log
    Code:
    2013-05-03 15:29:49,227 [1367609389] Warning : Printing stored debug messages prior to our daemonization
    2013-05-03 15:29:49,234 [1367609389] Warning : IOError(2, 'No such file or directory')
    But now I am stucked.
    Anyoone has a hint on were I shuld look to find the problem ?

  2. #2
    Administrator
    Join Date
    Dec 2011
    Posts
    278

    Re: SnmpBooster, genDevConfig Pending STATE

    Hmm.. No such file or directory error is a give away that something is wrong.

    Run Shinken in debug mode and see if you can find out where this error is occurring.

    Once that is fixed... We can move on to other steps.

    You can also see if there are any snmp requests being issued, but running tcmpdump or wireshark and looking for traffic destined to the device you want to monitor (or simply filtering for snmp protocol).

    Cheers,

    xkilian

  3. #3
    Junior Member
    Join Date
    May 2013
    Posts
    10

    Re: SnmpBooster, genDevConfig Pending STATE

    I added "log_level=DEBUG" to the following .ini to enable the debugging.

    brokerd.ini
    pollerd.ini
    reactionnerd.ini
    receiverd.ini
    schedulerd.ini

    I checked my log files.

    arbiter.log, Nothing special, IOError is gone.
    brokerd.log, Nothing special
    reactionnerd.log, Nothing special
    receiverd.log, Nothing special
    schedulerd.log, Nothing special
    skonfd.log, Nothing special

    pollerd.log (There is am message about a Mapping not done and a warning about worker 1 going down)
    Code:
    2013-05-04 14:20:55,894 [1367691655] Info :  [poller-1] Got module: snmp_poller
    2013-05-04 14:20:56,058 [1367691656] Info :  [SnmpBooster] Get a snmp poller module for plugin SnmpBooster
    2013-05-04 14:20:56,058 [1367691656] Info :  Trying to init module: SnmpBooster
    2013-05-04 14:20:56,059 [1367691656] Info :  [SnmpBooster] Initialization of the SNMP Booster 1.0
    2013-05-04 14:20:56,144 [1367691656] Info :  I correctly loaded the modules: [SnmpBooster]
    2013-05-04 14:20:56,146 [1367691656] Info :  [poller-1] Allocating new fork Worker: 0
    2013-05-04 14:20:56,150 [1367691656] Info :  [poller-1] Allocating new snmp_poller Worker: 1
    2013-05-04 14:20:56,151 [1367691656] Debug :  Loop turn
    2013-05-04 14:20:56,157 [1367691656] Info :  [SnmpBooster] Module SNMP Booster started!
    2013-05-04 14:20:57,152 [1367691657] Debug :  ========================
    2013-05-04 14:20:57,153 [1367691657] Debug :  [0][scheduler-1][fork] Stats: Workers:0 (Queued:0 Total ReturnWait:0)
    2013-05-04 14:20:57,154 [1367691657] Debug :  [0][scheduler-1][snmp_poller] Stats: Workers:1 (Queued:0 TotalReturnWait:0)
    2013-05-04 14:20:57,154 [1367691657] Debug :  Wait ratio: 1.000000
    2013-05-04 14:20:57,155 [1367691657] Debug :  Ask actions to 0, got 1
    2013-05-04 14:20:57,157 [1367691657] Debug :  Loop turn
    2013-05-04 14:20:57,264 [1367691657] Debug :  [SnmpBooster] Mapping not done : 192.168.254.253,FastEthernet0_19
    2013-05-04 14:20:58,157 [1367691658] Debug :  ========================
    2013-05-04 14:20:58,158 [1367691658] Warning : [poller-1] The worker 1 goes down unexpectedly!
    nagios.log (messages in 3 copies is it normal ?)
    Code:
    [1367691654] Info :  Shinken 1.2.4
    [1367691654] Info :  Shinken 1.2.4
    [1367691654] Info :  Shinken 1.2.4
    [1367691654] Info :  Copyright (c) 2009-2011:
    [1367691654] Info :  Copyright (c) 2009-2011:
    [1367691654] Info :  Copyright (c) 2009-2011:
    [1367691654] Info :  Gabes Jean (naparuba@gmail.com)
    [1367691654] Info :  Gabes Jean (naparuba@gmail.com)
    [1367691654] Info :  Gabes Jean (naparuba@gmail.com)
    [1367691654] Info :  Gerhard Lausser, Gerhard.Lausser@consol.de
    [1367691654] Info :  Gerhard Lausser, Gerhard.Lausser@consol.de
    [1367691654] Info :  Gerhard Lausser, Gerhard.Lausser@consol.de
    [1367691654] Info :  Gregory Starck, g.starck@gmail.com
    [1367691654] Info :  Gregory Starck, g.starck@gmail.com
    [1367691654] Info :  Gregory Starck, g.starck@gmail.com
    [1367691654] Info :  Hartmut Goebel, h.goebel@goebel-consult.de
    [1367691654] Info :  Hartmut Goebel, h.goebel@goebel-consult.de
    [1367691654] Info :  Hartmut Goebel, h.goebel@goebel-consult.de
    [1367691654] Info :  License: AGPL
    [1367691654] Info :  License: AGPL
    [1367691654] Info :  License: AGPL
    [1367691654] Info :  [Broker] Using working directory: /usr/local/shinken/var
    [1367691654] Info :  Trying to initialize additional groups for the daemon
    [1367691654] Info :  Trying to initialize additional groups for the daemon
    [1367691654] Info :  Trying to initialize additional groups for the daemon
    [1367691654] Warning : Stale pidfile exists (no or invalid or unreadable content). Reusing it.
    [1367691654] Warning : Stale pidfile exists (no or invalid or unreadable content). Reusing it.
    [1367691654] Warning : Stale pidfile exists (no or invalid or unreadable content). Reusing it.
    [1367691654] Info :  Initializing Pyro connection with host:0.0.0.0 port:7771 ssl:False
    [1367691654] Info :  Initializing Pyro connection with host:0.0.0.0 port:7769 ssl:False
    [1367691654] Info :  Initializing Pyro connection with host:0.0.0.0 port:7772 ssl:False
    
    ....
    
    [1367691655] Info :  [Arbiter] Serializing the configurations...
    [1367691655] Info :  Configuration Loaded
    [1367691655] Info :  Trying to initialize additional groups for the daemon
    [1367691655] Warning : Stale pidfile exists (no or invalid or unreadable content). Reusing it.
    [1367691655] Info :  Initializing Pyro connection with host:localhost port:7770 ssl:False
    [1367691655] Warning : Printing stored debug messages prior to our daemonization
    [1367691569] Error :  Failed sending configuration for receiver-1: connection lost: [Errno 104] Connection reset by peer
    [1367691569] Error :  [All] Dispatching failed for receiver receiver-1
    [1367691570] Error :  Failed sending configuration for receiver-1: connection failed
    [1367691570] Error :  [All] Dispatching failed for receiver receiver-1
    [1367691658] Warning : [poller-1] The worker 1 goes down unexpectedly!
    [1367691658] Info :  [poller-1] Allocating new fork Worker: 2
    [1367691658] Info :  [poller-1] Allocating new snmp_poller Worker: 3
    [1367691665] Warning : [poller-1] The worker 3 goes down unexpectedly!
    [1367692016] Warning : 14 actions never came back for the satellite 'poller-1'. I'm reenable them for polling
    [1367692076] Warning : 14 actions never came back for the satellite 'poller-1'. I'm reenable them for polling
    [1367692136] Warning : 14 actions never came back for the satellite 'poller-1'. I'm reenable them for polling
    [1367692197] Warning : 14 actions never came back for the satellite 'poller-1'. I'm reenable them for polling
    [1367692257] Warning : 14 actions never came back for the satellite 'poller-1'. I'm reenable them for polling
    [1367692317] Warning : 14 actions never came back for the satellite 'poller-1'. I'm reenable them for polling
    [1367692377] Warning : 14 actions never came back for the satellite 'poller-1'. I'm reenable them for polling
    [1367692437] Warning : 14 actions never came back for the satellite 'poller-1'. I'm reenable them for polling
    [1367692497] Warning : 14 actions never came back for the satellite 'poller-1'. I'm reenable them for polling
    [1367692523] SERVICE NOTIFICATION: admin;localhost;NetworkUsage;CRITICAL;notify-service-by-email;/bin/sh: 1: /usr/local/shinken/libexec/check_netint.pl: not found
    [1367692523] SERVICE NOTIFICATION: guest;localhost;NetworkUsage;CRITICAL;notify-service-by-email;/bin/sh: 1: /usr/local/shinken/libexec/check_netint.pl: not found
    [1367692524] Warning : The notification command '/usr/bin/printf "%b" "Shinken Notification\n\nNotification Type: PROBLEM\n\nService: NetworkUsage\nHost: localhost\nAddress: localhost\nState: CRITICAL\n\n
    Date/Time: 04-05-2013 Additional Info : /bin/sh: 1: /usr/local/shinken/libexec/check_netint.pl: not found\n" | /usr/bin/mail -s "** PROBLEM alert - localhost/NetworkUsage is CRITICAL **" shinken@localhost' raised an error (exit code=127): '/bin/sh: 1: /usr/bin/mail: not found'
    [1367692524] Warning : The notification command '/usr/bin/printf "%b" "Shinken Notification\n\nNotification Type: PROBLEM\n\nService: NetworkUsage\nHost: localhost\nAddress: localhost\nState: CRITICAL\n\n
    Date/Time: 04-05-2013 Additional Info : /bin/sh: 1: /usr/local/shinken/libexec/check_netint.pl: not found\n" | /usr/bin/mail -s "** PROBLEM alert - localhost/NetworkUsage is CRITICAL **" none' raised an error (exit code=127): '/bin/sh: 1: /usr/bin/mail: not found'
    [1367692557] Warning : 14 actions never came back for the satellite 'poller-1'. I'm reenable them for polling
    [1367692617] Warning : 14 actions never came back for the satellite 'poller-1'. I'm reenable them for polling
    schedulerd.log
    Code:
    2013-05-06 08:35:25,352 [1367843725] Warning : The notification command '/usr/bin/printf "%b" "Shinken Notification\n\nNotification Type: PROBLEM\n\nService: NetworkUsage\nHost: localhost\nAddress: localhost\nState: CRITICAL\n\nDate/Time: 06-05-2013 Additional Info : /bin/sh: 1: /usr/local/shinken/libexec/check_netint.pl: not found\n" | /usr/bin/mail -s "** PROBLEM alert - localhost/NetworkUsage is CRITICAL **" none' raised an error (exit code=127): '/bin/sh: 1: /usr/bin/mail: not found'
    2013-05-06 08:38:59,762 [1367843939] Warning : 14 actions never came back for the satellite 'poller-1'. I'm reenable them for polling

    grep SnmpBooster *
    Code:
    nagios.log:[1367843579] Info :  [SnmpBooster] Get a snmp poller module for plugin SnmpBooster
    nagios.log:[1367843579] Info :  Trying to init module: SnmpBooster
    nagios.log:[1367843579] Info :  [SnmpBooster] Initialization of the SNMP Booster 1.0
    nagios.log:[1367843579] Info :  I correctly loaded the modules: [SnmpBooster]
    nagios.log:[1367843576] Info :  My own modules: CommandFile,PickleRetentionArbiter,SnmpBooster
    nagios.log:[1367843577] Info :  [SnmpBooster] Get a snmp poller module for plugin SnmpBooster
    nagios.log:[1367843577] Info :  Trying to init module: SnmpBooster
    nagios.log:[1367843577] Info :  [SnmpBooster] Initialization of the SNMP Booster 1.0
    nagios.log:[1367843577] Info :  I correctly loaded the modules: [CommandFile,PickleRetentionArbiter,SnmpBooster]
    grep: pack_distribution.dat: Permission denied
    pollerd.log:2013-05-06 08:32:58,901 [1367843578] Debug :  Add module object {'configuration_errors': [], 'use': '', 'hash': '', 'name': '', 'tags': set([]), 'module_name': u'SnmpBooster', 'modules': [], 'customs': {}, 'plus': {}, 'datasource': u'/usr/local/shinken/etc/SNMP/', 'properties': {'daemons': ['poller', 'scheduler', 'arbiter'], 'phases': ['running', 'late_configuration', None], 'type': 'snmp_poller', 'external': False, 'worker_capable': True}, 'memcached_port': u'11211', 'memcached_host': u'127.0.0.1', 'configuration_warnings': [], 'module_type': u'snmp_poller', 'id': 2, 'imported_from': u'/usr/local/shinken/etc/shinken-specific.cfg'}
    pollerd.log:2013-05-06 08:32:59,119 [1367843579] Info :  [SnmpBooster] Get a snmp poller module for plugin SnmpBooster
    pollerd.log:2013-05-06 08:32:59,119 [1367843579] Info :  Trying to init module: SnmpBooster
    pollerd.log:2013-05-06 08:32:59,119 [1367843579] Info :  [SnmpBooster] Initialization of the SNMP Booster 1.0
    pollerd.log:2013-05-06 08:32:59,204 [1367843579] Info :  I correctly loaded the modules: [SnmpBooster]
    pollerd.log:2013-05-06 08:32:59,216 [1367843579] Info :  [SnmpBooster] Module SNMP Booster started!
    pollerd.log:2013-05-06 08:33:12,439 [1367843592] Debug :  [SnmpBooster] No old data : 192.168.254.253,None
    pollerd.log:2013-05-06 08:33:13,273 [1367843593] Info :  [SnmpBooster] Module SNMP Booster started!
    schedulerd.log:2013-05-06 08:32:58,777 [1367843578] Info :  [SnmpBooster] Get a snmp poller module for plugin SnmpBooster
    schedulerd.log:2013-05-06 08:32:58,777 [1367843578] Info :  Trying to init module: SnmpBooster
    schedulerd.log:2013-05-06 08:32:58,777 [1367843578] Info :  [SnmpBooster] Initialization of the SNMP Booster 1.0
    schedulerd.log:2013-05-06 08:32:58,871 [1367843578] Info :  I correctly loaded the modules: [PickleRetention,SnmpBooster]
    schedulerd.log:2013-05-06 08:32:58,877 [1367843578] Debug :  hook_point: SnmpBooster: False load_retention
    schedulerd.log:2013-05-06 08:32:58,877 [1367843578] Debug :  hook_point: SnmpBooster: False pre_scheduler_mod_start
    schedulerd.log:2013-05-06 08:32:59,895 [1367843579] Debug :  hook_point: SnmpBooster: True get_new_actions
    schedulerd.log:2013-05-06 08:33:01,009 [1367843581] Debug :  hook_point: SnmpBooster: True get_new_actions
    schedulerd.log:2013-05-06 08:33:02,011 [1367843582] Debug :  hook_point: SnmpBooster: True get_new_actions
    schedulerd.log:2013-05-06 08:33:03,013 [1367843583] Debug :  hook_point: SnmpBooster: True get_new_actions
    schedulerd.log:2013-05-06 08:33:04,015 [1367843584] Debug :  hook_point: SnmpBooster: True get_new_actions
    schedulerd.log:2013-05-06 08:33:05,016 [1367843585] Debug :  hook_point: SnmpBooster: True get_new_actions
    schedulerd.log:2013-05-06 08:33:06,018 [1367843586] Debug :  hook_point: SnmpBooster: True get_new_actions
    schedulerd.log:2013-05-06 08:33:07,020 [1367843587] Debug :  hook_point: SnmpBooster: True get_new_actions
    schedulerd.log:2013-05-06 08:33:08,022 [1367843588] Debug :  hook_point: SnmpBooster: True get_new_actions
    schedulerd.log:2013-05-06 08:33:09,023 [1367843589] Debug :  hook_point: SnmpBooster: True get_new_actions
    schedulerd.log:2013-05-06 08:33:10,025 [1367843590] Debug :  hook_point: SnmpBooster: True get_new_actions
    schedulerd.log:2013-05-06 08:33:11,027 [1367843591] Debug :  hook_point: SnmpBooster: True get_new_actions
    schedulerd.log:2013-05-06 08:33:12,029 [1367843592] Debug :  hook_point: SnmpBooster: True get_new_actions
    schedulerd.log:2013-05-06 08:33:13,030 [1367843593] Debug :  hook_point: SnmpBooster: True get_new_actions
    schedulerd.log:2013-05-06 08:33:14,032 [1367843594] Debug :  hook_point: SnmpBooster: True get_new_actions
    schedulerd.log:2013-05-06 08:33:15,034 [1367843595] Debug :  hook_point: SnmpBooster: True get_new_actions
    I did a tcpdump and I see only pings going out to the device no SNMP "I did a snmp walk and it did show up". It look like snmp_poller.py is no doing its job.


  4. #4
    Administrator
    Join Date
    Dec 2011
    Posts
    278

    Re: SnmpBooster, genDevConfig Pending STATE

    Hello Miraculor,

    Can you send me your configuration (via private message) for the command, host and shinken-specific.cfg for poller, scheduler, arbiter and the configuration generated by genDevConfig. i am intrigued by this, if something is wrong we are missing a debug message or not understanding what it is telling us.

    Thanks

    xkilian

  5. #5
    Administrator
    Join Date
    Dec 2011
    Posts
    278

    Re: SnmpBooster, genDevConfig Pending STATE

    I have just gone through a full re-install of Shinken + SnmpBooster here are some notes that may help:

    1 - The template file!

    Place it under shinken/packs/network/SnmpBooster/templates.cfg

    2 - The ini files
    Place them in /usr/local/shinken/etc/packs/network/SnmpBooster/

    3 - Datasource definition in shinken-specific.cfg
    datasource /usr/local/shinken/etc/packs/network/SnmpBooster/

    4 - Make sure your memcached is running and on the right port

    netstat -a | grep memcached

    If you need modify the IP or port do it in /etc/memcached.conf

    If your poller, scheduler or arbiter are not all running on the same server you must use a routable IP and not 127.0.0.1.

    5 - Create a host using genDevConfig and make sure it is read and accepted. (You can see that in your arbiter logs)

    6 - Only use the Defaults.*.ini files that you need. Remove the others.

    Cheers

  6. #6
    Junior Member
    Join Date
    May 2013
    Posts
    10

    Re: SnmpBooster, genDevConfig Pending STATE

    I tried a new install and I am still having the problems. I am missing something but I cannot put my finger on it.

    The server is a VMWARE virtual machine.
    I installed Ubuntu server 12.04 (32 Bit) an updated
    Code:
    sudo apt-get update
    sudo apt-get upgrade
    Installed Shinken dependencies:
    Code:
    sudo apt-get install pkg-config libcurl4-gnutls-dev python-setuptools
    sudo easy_install -U distribute
    Installed Shinken:
    Code:
    sudo curl -L http://install.shinken-monitoring.org | /bin/bash
    SnmpBooster dependencies:
    Code:
    sudo apt-get install python-pysnmp-common python-configobj python-memcache
    genDevConfig dependencies:
    Code:
    sudo apt-get install libsnmp-session-perl python-dateutil python-crypto memcachedb python-memcache pyro git
    Installed SnmpBooster & genDevConfig:
    Code:
    sudo passwd shinken
    su shinken
    cd /usr/local/shinken/shinken/modules
    wget https://raw.github.com/titilambert/s...snmp_poller.py
    exit
    
    cd /usr/local
    sudo git clone https://github.com/xkilian/genDevConfig
    sudo chown -R shinken genDevConfig
    sudo chgrp -R shinken genDevConfig
    
    su shinken
    mkdir /usr/local/shinken/etc/packs/network/SnmpBooster/
    cd /usr/local/shinken/etc/packs/network/SnmpBooster/
    export genDevConfig=/usr/local/genDevConfig
    export SHINKEN=/usr/local/shinken
    (mkdir $SHINKEN/etc/SNMP)
    cp $genDevConfig/sample-config/genConfig/Defaults* $SHINKEN/etc/packs/network/SnmpBooster/
    cp $genDevConfig/sample-config/genConfig/templates.cfg $SHINKEN/etc/packs/network/SnmpBooster/templates.cfg
    exit
    
    sudo chown shinken /usr/local/shinken/etc/shinken-specific.cfg
    sudo chgrp shinken /usr/local/shinken/etc/shinken-specific.cfg
    fixed the memcache port problem
    Code:
    sudo nano /etc/memcachedb.conf
    Code:
    # Default connection port is 11211
    #-p 21201
    -p 11211
    Modified shinken-specific.cfg
    Code:
    su shinken
    nano /usr/local/shinken/etc/shinken-specific.cfg
    Code:
    define poller {
     poller_name poller-1
     data_timeout 120
     check_interval 60
     polling_interval 1
     modules SnmpBooster
     processes_by_worker 256
     manage_sub_realms 0
     max_workers 0
     timeout 3
     address localhost
     realm All
     max_check_attempts 3
     min_workers 0
     port 7771
    }
    
    define realm {
     default 1
     realm_name All
    }
    
    define reactionner {
     data_timeout 120   
     check_interval 60
     polling_interval 1
     modules   
     reactionner_name reactionner-1
     manage_sub_realms 0     
     max_workers 15           
     spare 0
     timeout 3
     address localhost  
     realm All        
     max_check_attempts 3   
     min_workers 1   
     port 7769
    }
    
    define broker {       
     broker_name broker-1
     data_timeout 120    
     check_interval 60
     modules Livestatus, Simple-log, WebUI,NPCDMOD
     port 7772       
     manage_sub_realms 1       
     spare 0
     timeout 3
     address localhost  
     realm All
     max_check_attempts 3  
     manage_arbiters 1
    }
    
    define arbiter {
     modules CommandFile,PickleRetentionArbiter,SnmpBooster
     spare 0
     address localhost
     port 7770
     arbiter_name Arbiter-Master
    }
    
    define scheduler {
     data_timeout 120
     check_interval 60
     weight 1
     skip_initial_broks 0
     modules ,PickleRetention,SnmpBooster
     spare 0
     timeout 3
     address localhost
     scheduler_name scheduler-1
     max_check_attempts 3
     realm All
     port 7768
    }
    
    define module {
    
       module_name     SnmpBooster
       module_type     snmp_poller
       datasource      /usr/local/shinken/etc/packs/network/SnmpBooster/  
       memcached_host    127.0.0.1
       memcached_port    11211
    }
    added the host with genDevConfig
    Code:
    cd /usr/local/shinken/etc/hosts
    /usr/local/genDevConfig/util/genDevConfig 192.168.254.253 –showunused -c public
    sudo /etc/init.d/shinken restart
    Checked memcached
    Code:
    sudo netstat -e | grep 11211
    
    tcp    0   0 localhost:11211     localhost:47949     ESTABLISHED memcachedb 9396    
    tcp    0   0 localhost:47944     localhost:11211     ESTABLISHED shinken  8162    
    tcp    0   0 localhost:11211     localhost:47944     ESTABLISHED memcachedb 9382    
    tcp    0   0 localhost:47949     localhost:11211     ESTABLISHED shinken  9395
    Checked the trafic going to the host
    Code:
    sudo tcpdump host 192.168.254.253
    ONLY ICMP

    Changed the log level to debug (log_level=DEBUG)
    Code:
    nano brokerd.ini
    nano schedulerd.ini 
    nano pollerd.ini 
    nano reactionnerd.ini
    nano receiverd.ini
    I see "IOError" messages in the log but it can't figure wat is causing it
    Code:
    /usr/local/shinken/var$ grep -B2 -A2 Error *.log
    arbiter.log-2013-05-13 15:53:49,933 [1368474829] Warning : Printing stored debug messages prior to our daemonization
    arbiter.log:2013-05-13 15:53:49,942 [1368474829] Warning : IOError(2, 'No such file or directory')
    --
    nagios.log-[1368474830] Info :  [reactionner-1] Connection OK with scheduler scheduler-1
    nagios.log-[1368474829] Info :  [broker-1] We have our arbiters: {0: {'broks': {}, 'last_connection': 0, 'name': u'Arbiter-Master'
    nagios.log:[1368474830] Warning : IOError(2, 'No such file or directory')
    nagios.log-[1368474830] Info :  [poller-1] Using max workers: 2
    nagios.log-[1368474830] Info :  [reactionner-1] Using max workers: 15
    --
    nagios.log-[1368474829] Info :  Initializing Pyro connection with host:localhost port:7770 ssl:False
    nagios.log-[1368474829] Warning : Printing stored debug messages prior to our daemonization
    nagios.log:[1368474829] Warning : IOError(2, 'No such file or directory')
    nagios.log-[1368474837] Warning : [poller-1] The worker 1 goes down unexpectedly!
    nagios.log-[1368474837] Info :  [poller-1] Allocating new fork Worker: 2
    --
    schedulerd.log-2013-05-13 15:53:50,100 [1368474830] Debug :  hook_point: PickleRetention: True load_retention
    schedulerd.log-2013-05-13 15:53:50,100 [1368474830] Debug :  [PickleRetentionGeneric]Reading from retention_file /tmp/retention.dat
    schedulerd.log:2013-05-13 15:53:50,100 [1368474830] Warning : IOError(2, 'No such file or directory')
    schedulerd.log-2013-05-13 15:53:50,100 [1368474830] Debug :  hook_point: SnmpBooster: False load_retention
    schedulerd.log-2013-05-13 15:53:50,100 [1368474830] Debug :  hook_point: PickleRetention: False pre_scheduler_mod_start
    What am I missing ?

  7. #7
    Administrator
    Join Date
    Dec 2011
    Posts
    278

    Re: SnmpBooster, genDevConfig Pending STATE

    You did not install:

    PySNMP

    easy_install pysnmp

    Cheers,

    xkilian

  8. #8
    Junior Member
    Join Date
    May 2013
    Posts
    10

    Re: SnmpBooster, genDevConfig Pending STATE

    You are right my procedure was missing PySNMP.
    I installed it but I still have the pending states and the IOError.

    When you did the full re-install of Shinken + SnmpBooster was it on Ubuntu ?
    Did you install something else ?
    This problem is rely bugging me.

    Thank you for taking the time to answer all my newb questions, I truly appreciate the all the help.

  9. #9
    Administrator
    Join Date
    Dec 2011
    Posts
    278

    Re: SnmpBooster, genDevConfig Pending STATE

    You installed all the PySNMP dependencies too?

    I also imagine you installed correctly the genDevConfig dependencies or else it would not execute correctly. (SNMP_Session, etc.)

    You need to see the traceback message in the logs to know what file could not be opened which gives IOERROR.

    Only keep the Default base template and say, Cisco and Avaya templates to start. Delete the others. It may be a bad template.

    If you did get grep in the $shinken/var directory for SnmpBooster, nothing interesting comes up?

    I installed it on Ubuntu 12.04 Server LTS. I am only using the base, cisco and avaya templates, I will have to see if it is one of the others causing trouble.

    There must be some problem that is not being caught correctly that needs to be improved upon. (error message..)

    Thanks


  10. #10
    Junior Member
    Join Date
    May 2013
    Posts
    10

    Re: SnmpBooster, genDevConfig Pending STATE

    After looking at the logs yesterday I started to add more logger.dedug inside snmp_poller.py to understand what is going on.

    My theory is that something is causing a error and snmp_poller.py is crashing before being able to send a request to the switch (Cisco 2960) causing the message "Warning : [poller-1] The worker 3 goes down unexpectedly!"
    in pollerd.log and the condition causing the problem is probably not being reported to the log. This is why I am adding logger.debug messages everywere in my snmp_poller.py.

    I am still trying to understand exactly how thing are working inside snmp_poller.py but this is what I figured:
    The SNMPService mapping never get done and there is No old data (logical because there is no snmp request going out).

    I am getting closer to the part of the script I think is responsible for sending the request.

    Code:
    2013-05-15 08:23:44,647 [1368620624] Debug :  [0][scheduler-1][snmp_poller] Stats: Workers:3 (Queued:0 TotalReturnWait:0)
    2013-05-15 08:23:44,648 [1368620624] Debug :  Wait ratio: 1.000000
    2013-05-15 08:23:44,649 [1368620624] Debug :  Ask actions to 0, got 1
    2013-05-15 08:23:44,650 [1368620624] Debug :  Loop turn
    2013-05-15 08:23:45,049 [1368620625] Debug :  [SnmpBooster] Launch async SNMP request
    2013-05-15 08:23:45,054 [1368620625] Debug :  [SnmpBooster] SNMPAsyncClient self.obj:<snmp_poller.SNMPHost object at 0x998002c>
    2013-05-15 08:23:45,054 [1368620625] Debug :  [SnmpBooster] Map SNMPService Host:192.168.254.253 Name:chassis.device-traffic serv_key:('Device-Traffic', None, None)
    2013-05-15 08:23:45,055 [1368620625] Debug :  [SnmpBooster] SNMPService mapping :True
    2013-05-15 08:23:45,055 [1368620625] Debug :  [SnmpBooster] No old data : 192.168.254.253,None
    2013-05-15 08:23:45,650 [1368620625] Debug :  ========================
    2013-05-15 08:23:45,651 [1368620625] Warning : [poller-1] The worker 3 goes down unexpectedly!
    I will continue to report on my progress. Thanks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •