I added "log_level=DEBUG" to the following .ini to enable the debugging.
brokerd.ini
pollerd.ini
reactionnerd.ini
receiverd.ini
schedulerd.ini
I checked my log files.
arbiter.log, Nothing special, IOError is gone.
brokerd.log, Nothing special
reactionnerd.log, Nothing special
receiverd.log, Nothing special
schedulerd.log, Nothing special
skonfd.log, Nothing special
pollerd.log (There is am message about a Mapping not done and a warning about worker 1 going down)
Code:
2013-05-04 14:20:55,894 [1367691655] Info : [poller-1] Got module: snmp_poller
2013-05-04 14:20:56,058 [1367691656] Info : [SnmpBooster] Get a snmp poller module for plugin SnmpBooster
2013-05-04 14:20:56,058 [1367691656] Info : Trying to init module: SnmpBooster
2013-05-04 14:20:56,059 [1367691656] Info : [SnmpBooster] Initialization of the SNMP Booster 1.0
2013-05-04 14:20:56,144 [1367691656] Info : I correctly loaded the modules: [SnmpBooster]
2013-05-04 14:20:56,146 [1367691656] Info : [poller-1] Allocating new fork Worker: 0
2013-05-04 14:20:56,150 [1367691656] Info : [poller-1] Allocating new snmp_poller Worker: 1
2013-05-04 14:20:56,151 [1367691656] Debug : Loop turn
2013-05-04 14:20:56,157 [1367691656] Info : [SnmpBooster] Module SNMP Booster started!
2013-05-04 14:20:57,152 [1367691657] Debug : ========================
2013-05-04 14:20:57,153 [1367691657] Debug : [0][scheduler-1][fork] Stats: Workers:0 (Queued:0 Total ReturnWait:0)
2013-05-04 14:20:57,154 [1367691657] Debug : [0][scheduler-1][snmp_poller] Stats: Workers:1 (Queued:0 TotalReturnWait:0)
2013-05-04 14:20:57,154 [1367691657] Debug : Wait ratio: 1.000000
2013-05-04 14:20:57,155 [1367691657] Debug : Ask actions to 0, got 1
2013-05-04 14:20:57,157 [1367691657] Debug : Loop turn
2013-05-04 14:20:57,264 [1367691657] Debug : [SnmpBooster] Mapping not done : 192.168.254.253,FastEthernet0_19
2013-05-04 14:20:58,157 [1367691658] Debug : ========================
2013-05-04 14:20:58,158 [1367691658] Warning : [poller-1] The worker 1 goes down unexpectedly!
nagios.log (messages in 3 copies is it normal ?)
Code:
[1367691654] Info : Shinken 1.2.4
[1367691654] Info : Shinken 1.2.4
[1367691654] Info : Shinken 1.2.4
[1367691654] Info : Copyright (c) 2009-2011:
[1367691654] Info : Copyright (c) 2009-2011:
[1367691654] Info : Copyright (c) 2009-2011:
[1367691654] Info : Gabes Jean (naparuba@gmail.com)
[1367691654] Info : Gabes Jean (naparuba@gmail.com)
[1367691654] Info : Gabes Jean (naparuba@gmail.com)
[1367691654] Info : Gerhard Lausser, Gerhard.Lausser@consol.de
[1367691654] Info : Gerhard Lausser, Gerhard.Lausser@consol.de
[1367691654] Info : Gerhard Lausser, Gerhard.Lausser@consol.de
[1367691654] Info : Gregory Starck, g.starck@gmail.com
[1367691654] Info : Gregory Starck, g.starck@gmail.com
[1367691654] Info : Gregory Starck, g.starck@gmail.com
[1367691654] Info : Hartmut Goebel, h.goebel@goebel-consult.de
[1367691654] Info : Hartmut Goebel, h.goebel@goebel-consult.de
[1367691654] Info : Hartmut Goebel, h.goebel@goebel-consult.de
[1367691654] Info : License: AGPL
[1367691654] Info : License: AGPL
[1367691654] Info : License: AGPL
[1367691654] Info : [Broker] Using working directory: /usr/local/shinken/var
[1367691654] Info : Trying to initialize additional groups for the daemon
[1367691654] Info : Trying to initialize additional groups for the daemon
[1367691654] Info : Trying to initialize additional groups for the daemon
[1367691654] Warning : Stale pidfile exists (no or invalid or unreadable content). Reusing it.
[1367691654] Warning : Stale pidfile exists (no or invalid or unreadable content). Reusing it.
[1367691654] Warning : Stale pidfile exists (no or invalid or unreadable content). Reusing it.
[1367691654] Info : Initializing Pyro connection with host:0.0.0.0 port:7771 ssl:False
[1367691654] Info : Initializing Pyro connection with host:0.0.0.0 port:7769 ssl:False
[1367691654] Info : Initializing Pyro connection with host:0.0.0.0 port:7772 ssl:False
....
[1367691655] Info : [Arbiter] Serializing the configurations...
[1367691655] Info : Configuration Loaded
[1367691655] Info : Trying to initialize additional groups for the daemon
[1367691655] Warning : Stale pidfile exists (no or invalid or unreadable content). Reusing it.
[1367691655] Info : Initializing Pyro connection with host:localhost port:7770 ssl:False
[1367691655] Warning : Printing stored debug messages prior to our daemonization
[1367691569] Error : Failed sending configuration for receiver-1: connection lost: [Errno 104] Connection reset by peer
[1367691569] Error : [All] Dispatching failed for receiver receiver-1
[1367691570] Error : Failed sending configuration for receiver-1: connection failed
[1367691570] Error : [All] Dispatching failed for receiver receiver-1
[1367691658] Warning : [poller-1] The worker 1 goes down unexpectedly!
[1367691658] Info : [poller-1] Allocating new fork Worker: 2
[1367691658] Info : [poller-1] Allocating new snmp_poller Worker: 3
[1367691665] Warning : [poller-1] The worker 3 goes down unexpectedly!
[1367692016] Warning : 14 actions never came back for the satellite 'poller-1'. I'm reenable them for polling
[1367692076] Warning : 14 actions never came back for the satellite 'poller-1'. I'm reenable them for polling
[1367692136] Warning : 14 actions never came back for the satellite 'poller-1'. I'm reenable them for polling
[1367692197] Warning : 14 actions never came back for the satellite 'poller-1'. I'm reenable them for polling
[1367692257] Warning : 14 actions never came back for the satellite 'poller-1'. I'm reenable them for polling
[1367692317] Warning : 14 actions never came back for the satellite 'poller-1'. I'm reenable them for polling
[1367692377] Warning : 14 actions never came back for the satellite 'poller-1'. I'm reenable them for polling
[1367692437] Warning : 14 actions never came back for the satellite 'poller-1'. I'm reenable them for polling
[1367692497] Warning : 14 actions never came back for the satellite 'poller-1'. I'm reenable them for polling
[1367692523] SERVICE NOTIFICATION: admin;localhost;NetworkUsage;CRITICAL;notify-service-by-email;/bin/sh: 1: /usr/local/shinken/libexec/check_netint.pl: not found
[1367692523] SERVICE NOTIFICATION: guest;localhost;NetworkUsage;CRITICAL;notify-service-by-email;/bin/sh: 1: /usr/local/shinken/libexec/check_netint.pl: not found
[1367692524] Warning : The notification command '/usr/bin/printf "%b" "Shinken Notification\n\nNotification Type: PROBLEM\n\nService: NetworkUsage\nHost: localhost\nAddress: localhost\nState: CRITICAL\n\n
Date/Time: 04-05-2013 Additional Info : /bin/sh: 1: /usr/local/shinken/libexec/check_netint.pl: not found\n" | /usr/bin/mail -s "** PROBLEM alert - localhost/NetworkUsage is CRITICAL **" shinken@localhost' raised an error (exit code=127): '/bin/sh: 1: /usr/bin/mail: not found'
[1367692524] Warning : The notification command '/usr/bin/printf "%b" "Shinken Notification\n\nNotification Type: PROBLEM\n\nService: NetworkUsage\nHost: localhost\nAddress: localhost\nState: CRITICAL\n\n
Date/Time: 04-05-2013 Additional Info : /bin/sh: 1: /usr/local/shinken/libexec/check_netint.pl: not found\n" | /usr/bin/mail -s "** PROBLEM alert - localhost/NetworkUsage is CRITICAL **" none' raised an error (exit code=127): '/bin/sh: 1: /usr/bin/mail: not found'
[1367692557] Warning : 14 actions never came back for the satellite 'poller-1'. I'm reenable them for polling
[1367692617] Warning : 14 actions never came back for the satellite 'poller-1'. I'm reenable them for polling
schedulerd.log
Code:
2013-05-06 08:35:25,352 [1367843725] Warning : The notification command '/usr/bin/printf "%b" "Shinken Notification\n\nNotification Type: PROBLEM\n\nService: NetworkUsage\nHost: localhost\nAddress: localhost\nState: CRITICAL\n\nDate/Time: 06-05-2013 Additional Info : /bin/sh: 1: /usr/local/shinken/libexec/check_netint.pl: not found\n" | /usr/bin/mail -s "** PROBLEM alert - localhost/NetworkUsage is CRITICAL **" none' raised an error (exit code=127): '/bin/sh: 1: /usr/bin/mail: not found'
2013-05-06 08:38:59,762 [1367843939] Warning : 14 actions never came back for the satellite 'poller-1'. I'm reenable them for polling
grep SnmpBooster *
Code:
nagios.log:[1367843579] Info : [SnmpBooster] Get a snmp poller module for plugin SnmpBooster
nagios.log:[1367843579] Info : Trying to init module: SnmpBooster
nagios.log:[1367843579] Info : [SnmpBooster] Initialization of the SNMP Booster 1.0
nagios.log:[1367843579] Info : I correctly loaded the modules: [SnmpBooster]
nagios.log:[1367843576] Info : My own modules: CommandFile,PickleRetentionArbiter,SnmpBooster
nagios.log:[1367843577] Info : [SnmpBooster] Get a snmp poller module for plugin SnmpBooster
nagios.log:[1367843577] Info : Trying to init module: SnmpBooster
nagios.log:[1367843577] Info : [SnmpBooster] Initialization of the SNMP Booster 1.0
nagios.log:[1367843577] Info : I correctly loaded the modules: [CommandFile,PickleRetentionArbiter,SnmpBooster]
grep: pack_distribution.dat: Permission denied
pollerd.log:2013-05-06 08:32:58,901 [1367843578] Debug : Add module object {'configuration_errors': [], 'use': '', 'hash': '', 'name': '', 'tags': set([]), 'module_name': u'SnmpBooster', 'modules': [], 'customs': {}, 'plus': {}, 'datasource': u'/usr/local/shinken/etc/SNMP/', 'properties': {'daemons': ['poller', 'scheduler', 'arbiter'], 'phases': ['running', 'late_configuration', None], 'type': 'snmp_poller', 'external': False, 'worker_capable': True}, 'memcached_port': u'11211', 'memcached_host': u'127.0.0.1', 'configuration_warnings': [], 'module_type': u'snmp_poller', 'id': 2, 'imported_from': u'/usr/local/shinken/etc/shinken-specific.cfg'}
pollerd.log:2013-05-06 08:32:59,119 [1367843579] Info : [SnmpBooster] Get a snmp poller module for plugin SnmpBooster
pollerd.log:2013-05-06 08:32:59,119 [1367843579] Info : Trying to init module: SnmpBooster
pollerd.log:2013-05-06 08:32:59,119 [1367843579] Info : [SnmpBooster] Initialization of the SNMP Booster 1.0
pollerd.log:2013-05-06 08:32:59,204 [1367843579] Info : I correctly loaded the modules: [SnmpBooster]
pollerd.log:2013-05-06 08:32:59,216 [1367843579] Info : [SnmpBooster] Module SNMP Booster started!
pollerd.log:2013-05-06 08:33:12,439 [1367843592] Debug : [SnmpBooster] No old data : 192.168.254.253,None
pollerd.log:2013-05-06 08:33:13,273 [1367843593] Info : [SnmpBooster] Module SNMP Booster started!
schedulerd.log:2013-05-06 08:32:58,777 [1367843578] Info : [SnmpBooster] Get a snmp poller module for plugin SnmpBooster
schedulerd.log:2013-05-06 08:32:58,777 [1367843578] Info : Trying to init module: SnmpBooster
schedulerd.log:2013-05-06 08:32:58,777 [1367843578] Info : [SnmpBooster] Initialization of the SNMP Booster 1.0
schedulerd.log:2013-05-06 08:32:58,871 [1367843578] Info : I correctly loaded the modules: [PickleRetention,SnmpBooster]
schedulerd.log:2013-05-06 08:32:58,877 [1367843578] Debug : hook_point: SnmpBooster: False load_retention
schedulerd.log:2013-05-06 08:32:58,877 [1367843578] Debug : hook_point: SnmpBooster: False pre_scheduler_mod_start
schedulerd.log:2013-05-06 08:32:59,895 [1367843579] Debug : hook_point: SnmpBooster: True get_new_actions
schedulerd.log:2013-05-06 08:33:01,009 [1367843581] Debug : hook_point: SnmpBooster: True get_new_actions
schedulerd.log:2013-05-06 08:33:02,011 [1367843582] Debug : hook_point: SnmpBooster: True get_new_actions
schedulerd.log:2013-05-06 08:33:03,013 [1367843583] Debug : hook_point: SnmpBooster: True get_new_actions
schedulerd.log:2013-05-06 08:33:04,015 [1367843584] Debug : hook_point: SnmpBooster: True get_new_actions
schedulerd.log:2013-05-06 08:33:05,016 [1367843585] Debug : hook_point: SnmpBooster: True get_new_actions
schedulerd.log:2013-05-06 08:33:06,018 [1367843586] Debug : hook_point: SnmpBooster: True get_new_actions
schedulerd.log:2013-05-06 08:33:07,020 [1367843587] Debug : hook_point: SnmpBooster: True get_new_actions
schedulerd.log:2013-05-06 08:33:08,022 [1367843588] Debug : hook_point: SnmpBooster: True get_new_actions
schedulerd.log:2013-05-06 08:33:09,023 [1367843589] Debug : hook_point: SnmpBooster: True get_new_actions
schedulerd.log:2013-05-06 08:33:10,025 [1367843590] Debug : hook_point: SnmpBooster: True get_new_actions
schedulerd.log:2013-05-06 08:33:11,027 [1367843591] Debug : hook_point: SnmpBooster: True get_new_actions
schedulerd.log:2013-05-06 08:33:12,029 [1367843592] Debug : hook_point: SnmpBooster: True get_new_actions
schedulerd.log:2013-05-06 08:33:13,030 [1367843593] Debug : hook_point: SnmpBooster: True get_new_actions
schedulerd.log:2013-05-06 08:33:14,032 [1367843594] Debug : hook_point: SnmpBooster: True get_new_actions
schedulerd.log:2013-05-06 08:33:15,034 [1367843595] Debug : hook_point: SnmpBooster: True get_new_actions
I did a tcpdump and I see only pings going out to the device no SNMP "I did a snmp walk and it did show up". It look like snmp_poller.py is no doing its job.