SnmpBooster, genDevConfig Pending STATE

**Miraculor** · 05-03-2013, 08:39 PM

Hello,

This is my first Shinken installation (Ubuntu 12.04 server in vmware) I am running into problem when trying to monitor my Cisco switch with SnmpBooster.

i followed both of theses procedures to install:
http://www.shinken-monitoring.org/wi...en_10min_start
http://www.shinken-monitoring.org/wi...booster_module

I fixed the Ubuntu 12.04 Memcached errors using the SnmpBooster Troubleshooting document:
http://www.shinken-monitoring.org/wi...roubleshooting

I was able to generate the host config file with genDevConfig and everything look ok with SnmpBooster but in the WebUI all SNMP checks for the switch are in PENDING state.

I turned on debugging, the only thing i can find in the logs is this:

nagios.log

Code:

...
[1367609389] Info :  [Arbiter] Serializing the configurations...
[1367609389] Info :  Configuration Loaded
[1367609389] Info :  Trying to initialize additional groups for the daemon
[1367609389] Warning : Stale pidfile exists (no or invalid or unreadable content). Reusing it.
[1367609389] Info :  Initializing Pyro connection with host:localhost port:7770 ssl:False
[1367609389] Warning : Printing stored debug messages prior to our daemonization
[1367609389] Warning : IOError(2, &#039;No such file or directory&#039;)
[1367609391] Warning : [poller-1] The worker 1 goes down unexpectedly!
[1367609391] Info :  [poller-1] Allocating new fork Worker: 2
[1367609391] Info :  [poller-1] Allocating new snmp_poller Worker: 3
[1367609395] Warning : [poller-1] The worker 3 goes down unexpectedly!
...
1367610772] Warning : 14 actions never came back for the satellite &#039;poller-1&#039;. I&#039;m reenable them for polling
[1367610832] Warning : 14 actions never came back for the satellite &#039;poller-1&#039;. I&#039;m reenable them for polling
[1367610893] Warning : 14 actions never came back for the satellite &#039;poller-1&#039;. I&#039;m reenable them for polling
[1367610928] Warning : [reactionner-1] The worker 0 goes down unexpectedly!
[1367610928] Info :  [reactionner-1] Allocating new fork Worker: 1
[1367610953] Warning : 14 actions never came back for the satellite &#039;poller-1&#039;. I&#039;m reenable them for polling
[1367611013] Warning : 14 actions never came back for the satellite &#039;poller-1&#039;. I&#039;m reenable them for polling
[1367611073] Warning : 14 actions never came back for the satellite &#039;poller-1&#039;. I&#039;m reenable them for polling
...

arbiter.log

Code:

2013-05-03 15:29:49,227 [1367609389] Warning : Printing stored debug messages prior to our daemonization
2013-05-03 15:29:49,234 [1367609389] Warning : IOError(2, &#039;No such file or directory&#039;)

But now I am stucked.
Anyoone has a hint on were I shuld look to find the problem ?

**xkilian** · 05-03-2013, 10:04 PM

Hmm.. No such file or directory error is a give away that something is wrong.

Run Shinken in debug mode and see if you can find out where this error is occurring.

Once that is fixed... We can move on to other steps.

You can also see if there are any snmp requests being issued, but running tcmpdump or wireshark and looking for traffic destined to the device you want to monitor (or simply filtering for snmp protocol).

Cheers,

xkilian

**Miraculor** · 05-06-2013, 01:09 PM

I added "log_level=DEBUG" to the following .ini to enable the debugging.

brokerd.ini
pollerd.ini
reactionnerd.ini
receiverd.ini
schedulerd.ini

I checked my log files.

arbiter.log, Nothing special, IOError is gone.
brokerd.log, Nothing special
reactionnerd.log, Nothing special
receiverd.log, Nothing special
schedulerd.log, Nothing special
skonfd.log, Nothing special

pollerd.log (There is am message about a Mapping not done and a warning about worker 1 going down)

Code:

2013-05-04 14:20:55,894 [1367691655] Info :  [poller-1] Got module: snmp_poller
2013-05-04 14:20:56,058 [1367691656] Info :  [SnmpBooster] Get a snmp poller module for plugin SnmpBooster
2013-05-04 14:20:56,058 [1367691656] Info :  Trying to init module: SnmpBooster
2013-05-04 14:20:56,059 [1367691656] Info :  [SnmpBooster] Initialization of the SNMP Booster 1.0
2013-05-04 14:20:56,144 [1367691656] Info :  I correctly loaded the modules: [SnmpBooster]
2013-05-04 14:20:56,146 [1367691656] Info :  [poller-1] Allocating new fork Worker: 0
2013-05-04 14:20:56,150 [1367691656] Info :  [poller-1] Allocating new snmp_poller Worker: 1
2013-05-04 14:20:56,151 [1367691656] Debug :  Loop turn
2013-05-04 14:20:56,157 [1367691656] Info :  [SnmpBooster] Module SNMP Booster started!
2013-05-04 14:20:57,152 [1367691657] Debug :  ========================
2013-05-04 14:20:57,153 [1367691657] Debug :  [0][scheduler-1][fork] Stats: Workers:0 (Queued:0 Total ReturnWait:0)
2013-05-04 14:20:57,154 [1367691657] Debug :  [0][scheduler-1][snmp_poller] Stats: Workers:1 (Queued:0 TotalReturnWait:0)
2013-05-04 14:20:57,154 [1367691657] Debug :  Wait ratio: 1.000000
2013-05-04 14:20:57,155 [1367691657] Debug :  Ask actions to 0, got 1
2013-05-04 14:20:57,157 [1367691657] Debug :  Loop turn
2013-05-04 14:20:57,264 [1367691657] Debug :  [SnmpBooster] Mapping not done : 192.168.254.253,FastEthernet0_19
2013-05-04 14:20:58,157 [1367691658] Debug :  ========================
2013-05-04 14:20:58,158 [1367691658] Warning : [poller-1] The worker 1 goes down unexpectedly!

nagios.log (messages in 3 copies is it normal ?)

Code:

[1367691654] Info :  Shinken 1.2.4
[1367691654] Info :  Shinken 1.2.4
[1367691654] Info :  Shinken 1.2.4
[1367691654] Info :  Copyright (c) 2009-2011:
[1367691654] Info :  Copyright (c) 2009-2011:
[1367691654] Info :  Copyright (c) 2009-2011:
[1367691654] Info :  Gabes Jean (naparuba@gmail.com)
[1367691654] Info :  Gabes Jean (naparuba@gmail.com)
[1367691654] Info :  Gabes Jean (naparuba@gmail.com)
[1367691654] Info :  Gerhard Lausser, Gerhard.Lausser@consol.de
[1367691654] Info :  Gerhard Lausser, Gerhard.Lausser@consol.de
[1367691654] Info :  Gerhard Lausser, Gerhard.Lausser@consol.de
[1367691654] Info :  Gregory Starck, g.starck@gmail.com
[1367691654] Info :  Gregory Starck, g.starck@gmail.com
[1367691654] Info :  Gregory Starck, g.starck@gmail.com
[1367691654] Info :  Hartmut Goebel, h.goebel@goebel-consult.de
[1367691654] Info :  Hartmut Goebel, h.goebel@goebel-consult.de
[1367691654] Info :  Hartmut Goebel, h.goebel@goebel-consult.de
[1367691654] Info :  License: AGPL
[1367691654] Info :  License: AGPL
[1367691654] Info :  License: AGPL
[1367691654] Info :  [Broker] Using working directory: /usr/local/shinken/var
[1367691654] Info :  Trying to initialize additional groups for the daemon
[1367691654] Info :  Trying to initialize additional groups for the daemon
[1367691654] Info :  Trying to initialize additional groups for the daemon
[1367691654] Warning : Stale pidfile exists (no or invalid or unreadable content). Reusing it.
[1367691654] Warning : Stale pidfile exists (no or invalid or unreadable content). Reusing it.
[1367691654] Warning : Stale pidfile exists (no or invalid or unreadable content). Reusing it.
[1367691654] Info :  Initializing Pyro connection with host:0.0.0.0 port:7771 ssl:False
[1367691654] Info :  Initializing Pyro connection with host:0.0.0.0 port:7769 ssl:False
[1367691654] Info :  Initializing Pyro connection with host:0.0.0.0 port:7772 ssl:False

....

[1367691655] Info :  [Arbiter] Serializing the configurations...
[1367691655] Info :  Configuration Loaded
[1367691655] Info :  Trying to initialize additional groups for the daemon
[1367691655] Warning : Stale pidfile exists (no or invalid or unreadable content). Reusing it.
[1367691655] Info :  Initializing Pyro connection with host:localhost port:7770 ssl:False
[1367691655] Warning : Printing stored debug messages prior to our daemonization
[1367691569] Error :  Failed sending configuration for receiver-1: connection lost: [Errno 104] Connection reset by peer
[1367691569] Error :  [All] Dispatching failed for receiver receiver-1
[1367691570] Error :  Failed sending configuration for receiver-1: connection failed
[1367691570] Error :  [All] Dispatching failed for receiver receiver-1
[1367691658] Warning : [poller-1] The worker 1 goes down unexpectedly!
[1367691658] Info :  [poller-1] Allocating new fork Worker: 2
[1367691658] Info :  [poller-1] Allocating new snmp_poller Worker: 3
[1367691665] Warning : [poller-1] The worker 3 goes down unexpectedly!
[1367692016] Warning : 14 actions never came back for the satellite &#039;poller-1&#039;. I&#039;m reenable them for polling
[1367692076] Warning : 14 actions never came back for the satellite &#039;poller-1&#039;. I&#039;m reenable them for polling
[1367692136] Warning : 14 actions never came back for the satellite &#039;poller-1&#039;. I&#039;m reenable them for polling
[1367692197] Warning : 14 actions never came back for the satellite &#039;poller-1&#039;. I&#039;m reenable them for polling
[1367692257] Warning : 14 actions never came back for the satellite &#039;poller-1&#039;. I&#039;m reenable them for polling
[1367692317] Warning : 14 actions never came back for the satellite &#039;poller-1&#039;. I&#039;m reenable them for polling
[1367692377] Warning : 14 actions never came back for the satellite &#039;poller-1&#039;. I&#039;m reenable them for polling
[1367692437] Warning : 14 actions never came back for the satellite &#039;poller-1&#039;. I&#039;m reenable them for polling
[1367692497] Warning : 14 actions never came back for the satellite &#039;poller-1&#039;. I&#039;m reenable them for polling
[1367692523] SERVICE NOTIFICATION: admin;localhost;NetworkUsage;CRITICAL;notify-service-by-email;/bin/sh: 1: /usr/local/shinken/libexec/check_netint.pl: not found
[1367692523] SERVICE NOTIFICATION: guest;localhost;NetworkUsage;CRITICAL;notify-service-by-email;/bin/sh: 1: /usr/local/shinken/libexec/check_netint.pl: not found
[1367692524] Warning : The notification command &#039;/usr/bin/printf &quot;%b&quot; &quot;Shinken Notification\n\nNotification Type: PROBLEM\n\nService: NetworkUsage\nHost: localhost\nAddress: localhost\nState: CRITICAL\n\n
Date/Time: 04-05-2013 Additional Info : /bin/sh: 1: /usr/local/shinken/libexec/check_netint.pl: not found\n&quot; | /usr/bin/mail -s &quot;** PROBLEM alert - localhost/NetworkUsage is CRITICAL **&quot; shinken@localhost&#039; raised an error (exit code=127): &#039;/bin/sh: 1: /usr/bin/mail: not found&#039;
[1367692524] Warning : The notification command &#039;/usr/bin/printf &quot;%b&quot; &quot;Shinken Notification\n\nNotification Type: PROBLEM\n\nService: NetworkUsage\nHost: localhost\nAddress: localhost\nState: CRITICAL\n\n
Date/Time: 04-05-2013 Additional Info : /bin/sh: 1: /usr/local/shinken/libexec/check_netint.pl: not found\n&quot; | /usr/bin/mail -s &quot;** PROBLEM alert - localhost/NetworkUsage is CRITICAL **&quot; none&#039; raised an error (exit code=127): &#039;/bin/sh: 1: /usr/bin/mail: not found&#039;
[1367692557] Warning : 14 actions never came back for the satellite &#039;poller-1&#039;. I&#039;m reenable them for polling
[1367692617] Warning : 14 actions never came back for the satellite &#039;poller-1&#039;. I&#039;m reenable them for polling

schedulerd.log

Code:

2013-05-06 08:35:25,352 [1367843725] Warning : The notification command &#039;/usr/bin/printf &quot;%b&quot; &quot;Shinken Notification\n\nNotification Type: PROBLEM\n\nService: NetworkUsage\nHost: localhost\nAddress: localhost\nState: CRITICAL\n\nDate/Time: 06-05-2013 Additional Info : /bin/sh: 1: /usr/local/shinken/libexec/check_netint.pl: not found\n&quot; | /usr/bin/mail -s &quot;** PROBLEM alert - localhost/NetworkUsage is CRITICAL **&quot; none&#039; raised an error (exit code=127): &#039;/bin/sh: 1: /usr/bin/mail: not found&#039;
2013-05-06 08:38:59,762 [1367843939] Warning : 14 actions never came back for the satellite &#039;poller-1&#039;. I&#039;m reenable them for polling

grep SnmpBooster *

Code:

nagios.log:[1367843579] Info :  [SnmpBooster] Get a snmp poller module for plugin SnmpBooster
nagios.log:[1367843579] Info :  Trying to init module: SnmpBooster
nagios.log:[1367843579] Info :  [SnmpBooster] Initialization of the SNMP Booster 1.0
nagios.log:[1367843579] Info :  I correctly loaded the modules: [SnmpBooster]
nagios.log:[1367843576] Info :  My own modules: CommandFile,PickleRetentionArbiter,SnmpBooster
nagios.log:[1367843577] Info :  [SnmpBooster] Get a snmp poller module for plugin SnmpBooster
nagios.log:[1367843577] Info :  Trying to init module: SnmpBooster
nagios.log:[1367843577] Info :  [SnmpBooster] Initialization of the SNMP Booster 1.0
nagios.log:[1367843577] Info :  I correctly loaded the modules: [CommandFile,PickleRetentionArbiter,SnmpBooster]
grep: pack_distribution.dat: Permission denied
pollerd.log:2013-05-06 08:32:58,901 [1367843578] Debug :  Add module object {&#039;configuration_errors&#039;: &#91;], &#039;use&#039;: &#039;&#039;, &#039;hash&#039;: &#039;&#039;, &#039;name&#039;: &#039;&#039;, &#039;tags&#039;: set(&#91;]), &#039;module_name&#039;: u&#039;SnmpBooster&#039;, &#039;modules&#039;: &#91;], &#039;customs&#039;: {}, &#039;plus&#039;: {}, &#039;datasource&#039;: u&#039;/usr/local/shinken/etc/SNMP/&#039;, &#039;properties&#039;: {&#039;daemons&#039;: &#91;&#039;poller&#039;, &#039;scheduler&#039;, &#039;arbiter&#039;], &#039;phases&#039;: &#91;&#039;running&#039;, &#039;late_configuration&#039;, None], &#039;type&#039;: &#039;snmp_poller&#039;, &#039;external&#039;: False, &#039;worker_capable&#039;: True}, &#039;memcached_port&#039;: u&#039;11211&#039;, &#039;memcached_host&#039;: u&#039;127.0.0.1&#039;, &#039;configuration_warnings&#039;: &#91;], &#039;module_type&#039;: u&#039;snmp_poller&#039;, &#039;id&#039;: 2, &#039;imported_from&#039;: u&#039;/usr/local/shinken/etc/shinken-specific.cfg&#039;}
pollerd.log:2013-05-06 08:32:59,119 [1367843579] Info :  [SnmpBooster] Get a snmp poller module for plugin SnmpBooster
pollerd.log:2013-05-06 08:32:59,119 [1367843579] Info :  Trying to init module: SnmpBooster
pollerd.log:2013-05-06 08:32:59,119 [1367843579] Info :  [SnmpBooster] Initialization of the SNMP Booster 1.0
pollerd.log:2013-05-06 08:32:59,204 [1367843579] Info :  I correctly loaded the modules: [SnmpBooster]
pollerd.log:2013-05-06 08:32:59,216 [1367843579] Info :  [SnmpBooster] Module SNMP Booster started!
pollerd.log:2013-05-06 08:33:12,439 [1367843592] Debug :  [SnmpBooster] No old data : 192.168.254.253,None
pollerd.log:2013-05-06 08:33:13,273 [1367843593] Info :  [SnmpBooster] Module SNMP Booster started!
schedulerd.log:2013-05-06 08:32:58,777 [1367843578] Info :  [SnmpBooster] Get a snmp poller module for plugin SnmpBooster
schedulerd.log:2013-05-06 08:32:58,777 [1367843578] Info :  Trying to init module: SnmpBooster
schedulerd.log:2013-05-06 08:32:58,777 [1367843578] Info :  [SnmpBooster] Initialization of the SNMP Booster 1.0
schedulerd.log:2013-05-06 08:32:58,871 [1367843578] Info :  I correctly loaded the modules: [PickleRetention,SnmpBooster]
schedulerd.log:2013-05-06 08:32:58,877 [1367843578] Debug :  hook_point: SnmpBooster: False load_retention
schedulerd.log:2013-05-06 08:32:58,877 [1367843578] Debug :  hook_point: SnmpBooster: False pre_scheduler_mod_start
schedulerd.log:2013-05-06 08:32:59,895 [1367843579] Debug :  hook_point: SnmpBooster: True get_new_actions
schedulerd.log:2013-05-06 08:33:01,009 [1367843581] Debug :  hook_point: SnmpBooster: True get_new_actions
schedulerd.log:2013-05-06 08:33:02,011 [1367843582] Debug :  hook_point: SnmpBooster: True get_new_actions
schedulerd.log:2013-05-06 08:33:03,013 [1367843583] Debug :  hook_point: SnmpBooster: True get_new_actions
schedulerd.log:2013-05-06 08:33:04,015 [1367843584] Debug :  hook_point: SnmpBooster: True get_new_actions
schedulerd.log:2013-05-06 08:33:05,016 [1367843585] Debug :  hook_point: SnmpBooster: True get_new_actions
schedulerd.log:2013-05-06 08:33:06,018 [1367843586] Debug :  hook_point: SnmpBooster: True get_new_actions
schedulerd.log:2013-05-06 08:33:07,020 [1367843587] Debug :  hook_point: SnmpBooster: True get_new_actions
schedulerd.log:2013-05-06 08:33:08,022 [1367843588] Debug :  hook_point: SnmpBooster: True get_new_actions
schedulerd.log:2013-05-06 08:33:09,023 [1367843589] Debug :  hook_point: SnmpBooster: True get_new_actions
schedulerd.log:2013-05-06 08:33:10,025 [1367843590] Debug :  hook_point: SnmpBooster: True get_new_actions
schedulerd.log:2013-05-06 08:33:11,027 [1367843591] Debug :  hook_point: SnmpBooster: True get_new_actions
schedulerd.log:2013-05-06 08:33:12,029 [1367843592] Debug :  hook_point: SnmpBooster: True get_new_actions
schedulerd.log:2013-05-06 08:33:13,030 [1367843593] Debug :  hook_point: SnmpBooster: True get_new_actions
schedulerd.log:2013-05-06 08:33:14,032 [1367843594] Debug :  hook_point: SnmpBooster: True get_new_actions
schedulerd.log:2013-05-06 08:33:15,034 [1367843595] Debug :  hook_point: SnmpBooster: True get_new_actions

I did a tcpdump and I see only pings going out to the device no SNMP "I did a snmp walk and it did show up". It look like snmp_poller.py is no doing its job.

**xkilian** · 05-06-2013, 01:15 PM

Hello Miraculor,

Can you send me your configuration (via private message) for the command, host and shinken-specific.cfg for poller, scheduler, arbiter and the configuration generated by genDevConfig. i am intrigued by this, if something is wrong we are missing a debug message or not understanding what it is telling us.

Thanks

xkilian

**xkilian** · 05-07-2013, 07:05 PM

I have just gone through a full re-install of Shinken + SnmpBooster here are some notes that may help:

1 - The template file!

Place it under shinken/packs/network/SnmpBooster/templates.cfg

2 - The ini files
Place them in /usr/local/shinken/etc/packs/network/SnmpBooster/

3 - Datasource definition in shinken-specific.cfg
datasource /usr/local/shinken/etc/packs/network/SnmpBooster/

4 - Make sure your memcached is running and on the right port

netstat -a | grep memcached

If you need modify the IP or port do it in /etc/memcached.conf

If your poller, scheduler or arbiter are not all running on the same server you must use a routable IP and not 127.0.0.1.

5 - Create a host using genDevConfig and make sure it is read and accepted. (You can see that in your arbiter logs)

6 - Only use the Defaults.*.ini files that you need. Remove the others.

Cheers

**Miraculor** · 05-13-2013, 08:22 PM

I tried a new install and I am still having the problems. I am missing something but I cannot put my finger on it.

The server is a VMWARE virtual machine.
I installed Ubuntu server 12.04 (32 Bit) an updated

Code:

sudo apt-get update
sudo apt-get upgrade

Installed Shinken dependencies:

Code:

sudo apt-get install pkg-config libcurl4-gnutls-dev python-setuptools
sudo easy_install -U distribute

Installed Shinken:

Code:

sudo curl -L http://install.shinken-monitoring.org | /bin/bash

SnmpBooster dependencies:

Code:

sudo apt-get install python-pysnmp-common python-configobj python-memcache

genDevConfig dependencies:

Code:

sudo apt-get install libsnmp-session-perl python-dateutil python-crypto memcachedb python-memcache pyro git

Installed SnmpBooster & genDevConfig:

Code:

sudo passwd shinken
su shinken
cd /usr/local/shinken/shinken/modules
wget https://raw.github.com/titilambert/s...snmp_poller.py
exit

cd /usr/local
sudo git clone https://github.com/xkilian/genDevConfig
sudo chown -R shinken genDevConfig
sudo chgrp -R shinken genDevConfig

su shinken
mkdir /usr/local/shinken/etc/packs/network/SnmpBooster/
cd /usr/local/shinken/etc/packs/network/SnmpBooster/
export genDevConfig=/usr/local/genDevConfig
export SHINKEN=/usr/local/shinken
(mkdir $SHINKEN/etc/SNMP)
cp $genDevConfig/sample-config/genConfig/Defaults* $SHINKEN/etc/packs/network/SnmpBooster/
cp $genDevConfig/sample-config/genConfig/templates.cfg $SHINKEN/etc/packs/network/SnmpBooster/templates.cfg
exit

sudo chown shinken /usr/local/shinken/etc/shinken-specific.cfg
sudo chgrp shinken /usr/local/shinken/etc/shinken-specific.cfg

fixed the memcache port problem

Code:

sudo nano /etc/memcachedb.conf

Code:

# Default connection port is 11211
#-p 21201
-p 11211

Modified shinken-specific.cfg

Code:

su shinken
nano /usr/local/shinken/etc/shinken-specific.cfg

Code:

define poller {
 poller_name poller-1
 data_timeout 120
 check_interval 60
 polling_interval 1
 modules SnmpBooster
 processes_by_worker 256
 manage_sub_realms 0
 max_workers 0
 timeout 3
 address localhost
 realm All
 max_check_attempts 3
 min_workers 0
 port 7771
}

define realm {
 default 1
 realm_name All
}

define reactionner {
 data_timeout 120   
 check_interval 60
 polling_interval 1
 modules   
 reactionner_name reactionner-1
 manage_sub_realms 0     
 max_workers 15           
 spare 0
 timeout 3
 address localhost  
 realm All        
 max_check_attempts 3   
 min_workers 1   
 port 7769
}

define broker {       
 broker_name broker-1
 data_timeout 120    
 check_interval 60
 modules Livestatus, Simple-log, WebUI,NPCDMOD
 port 7772       
 manage_sub_realms 1       
 spare 0
 timeout 3
 address localhost  
 realm All
 max_check_attempts 3  
 manage_arbiters 1
}

define arbiter {
 modules CommandFile,PickleRetentionArbiter,SnmpBooster
 spare 0
 address localhost
 port 7770
 arbiter_name Arbiter-Master
}

define scheduler {
 data_timeout 120
 check_interval 60
 weight 1
 skip_initial_broks 0
 modules ,PickleRetention,SnmpBooster
 spare 0
 timeout 3
 address localhost
 scheduler_name scheduler-1
 max_check_attempts 3
 realm All
 port 7768
}

define module {

   module_name     SnmpBooster
   module_type     snmp_poller
   datasource      /usr/local/shinken/etc/packs/network/SnmpBooster/  
   memcached_host    127.0.0.1
   memcached_port    11211
}

added the host with genDevConfig

Code:

cd /usr/local/shinken/etc/hosts
/usr/local/genDevConfig/util/genDevConfig 192.168.254.253 –showunused -c public
sudo /etc/init.d/shinken restart

Checked memcached

Code:

sudo netstat -e | grep 11211

tcp    0   0 localhost:11211     localhost:47949     ESTABLISHED memcachedb 9396    
tcp    0   0 localhost:47944     localhost:11211     ESTABLISHED shinken  8162    
tcp    0   0 localhost:11211     localhost:47944     ESTABLISHED memcachedb 9382    
tcp    0   0 localhost:47949     localhost:11211     ESTABLISHED shinken  9395

Checked the trafic going to the host

Code:

sudo tcpdump host 192.168.254.253

ONLY ICMP

Changed the log level to debug (log_level=DEBUG)

Code:

nano brokerd.ini
nano schedulerd.ini 
nano pollerd.ini 
nano reactionnerd.ini
nano receiverd.ini

I see "IOError" messages in the log but it can't figure wat is causing it

Code:

/usr/local/shinken/var$ grep -B2 -A2 Error *.log
arbiter.log-2013-05-13 15:53:49,933 [1368474829] Warning : Printing stored debug messages prior to our daemonization
arbiter.log:2013-05-13 15:53:49,942 [1368474829] Warning : IOError(2, &#039;No such file or directory&#039;)
--
nagios.log-[1368474830] Info :  [reactionner-1] Connection OK with scheduler scheduler-1
nagios.log-[1368474829] Info :  [broker-1] We have our arbiters: {0: {&#039;broks&#039;: {}, &#039;last_connection&#039;: 0, &#039;name&#039;: u&#039;Arbiter-Master&#039;
nagios.log:[1368474830] Warning : IOError(2, &#039;No such file or directory&#039;)
nagios.log-[1368474830] Info :  [poller-1] Using max workers: 2
nagios.log-[1368474830] Info :  [reactionner-1] Using max workers: 15
--
nagios.log-[1368474829] Info :  Initializing Pyro connection with host:localhost port:7770 ssl:False
nagios.log-[1368474829] Warning : Printing stored debug messages prior to our daemonization
nagios.log:[1368474829] Warning : IOError(2, &#039;No such file or directory&#039;)
nagios.log-[1368474837] Warning : [poller-1] The worker 1 goes down unexpectedly!
nagios.log-[1368474837] Info :  [poller-1] Allocating new fork Worker: 2
--
schedulerd.log-2013-05-13 15:53:50,100 [1368474830] Debug :  hook_point: PickleRetention: True load_retention
schedulerd.log-2013-05-13 15:53:50,100 [1368474830] Debug :  [PickleRetentionGeneric]Reading from retention_file /tmp/retention.dat
schedulerd.log:2013-05-13 15:53:50,100 [1368474830] Warning : IOError(2, &#039;No such file or directory&#039;)
schedulerd.log-2013-05-13 15:53:50,100 [1368474830] Debug :  hook_point: SnmpBooster: False load_retention
schedulerd.log-2013-05-13 15:53:50,100 [1368474830] Debug :  hook_point: PickleRetention: False pre_scheduler_mod_start

What am I missing ?

**xkilian** · 05-13-2013, 11:51 PM

You did not install:

PySNMP

easy_install pysnmp

Cheers,

xkilian

**Miraculor** · 05-14-2013, 01:45 PM

You are right my procedure was missing PySNMP.
I installed it but I still have the pending states and the IOError.

When you did the full re-install of Shinken + SnmpBooster was it on Ubuntu ?
Did you install something else ?
This problem is rely bugging me.

Thank you for taking the time to answer all my newb questions, I truly appreciate the all the help.

**xkilian** · 05-14-2013, 06:02 PM

You installed all the PySNMP dependencies too?

I also imagine you installed correctly the genDevConfig dependencies or else it would not execute correctly. (SNMP_Session, etc.)

You need to see the traceback message in the logs to know what file could not be opened which gives IOERROR.

Only keep the Default base template and say, Cisco and Avaya templates to start. Delete the others. It may be a bad template.

If you did get grep in the $shinken/var directory for SnmpBooster, nothing interesting comes up?

I installed it on Ubuntu 12.04 Server LTS. I am only using the base, cisco and avaya templates, I will have to see if it is one of the others causing trouble.

There must be some problem that is not being caught correctly that needs to be improved upon. (error message..)

Thanks

**Miraculor** · 05-15-2013, 12:31 PM

After looking at the logs yesterday I started to add more logger.dedug inside snmp_poller.py to understand what is going on.

My theory is that something is causing a error and snmp_poller.py is crashing before being able to send a request to the switch (Cisco 2960) causing the message "Warning : [poller-1] The worker 3 goes down unexpectedly!"
in pollerd.log and the condition causing the problem is probably not being reported to the log. This is why I am adding logger.debug messages everywere in my snmp_poller.py.

I am still trying to understand exactly how thing are working inside snmp_poller.py but this is what I figured:
The SNMPService mapping never get done and there is No old data (logical because there is no snmp request going out).

I am getting closer to the part of the script I think is responsible for sending the request.

Code:

2013-05-15 08:23:44,647 [1368620624] Debug :  [0][scheduler-1][snmp_poller] Stats: Workers:3 (Queued:0 TotalReturnWait:0)
2013-05-15 08:23:44,648 [1368620624] Debug :  Wait ratio: 1.000000
2013-05-15 08:23:44,649 [1368620624] Debug :  Ask actions to 0, got 1
2013-05-15 08:23:44,650 [1368620624] Debug :  Loop turn
2013-05-15 08:23:45,049 [1368620625] Debug :  [SnmpBooster] Launch async SNMP request
2013-05-15 08:23:45,054 [1368620625] Debug :  [SnmpBooster] SNMPAsyncClient self.obj:&lt;snmp_poller.SNMPHost object at 0x998002c&gt;
2013-05-15 08:23:45,054 [1368620625] Debug :  [SnmpBooster] Map SNMPService Host:192.168.254.253 Name:chassis.device-traffic serv_key:(&#039;Device-Traffic&#039;, None, None)
2013-05-15 08:23:45,055 [1368620625] Debug :  [SnmpBooster] SNMPService mapping :True
2013-05-15 08:23:45,055 [1368620625] Debug :  [SnmpBooster] No old data : 192.168.254.253,None
2013-05-15 08:23:45,650 [1368620625] Debug :  ========================
2013-05-15 08:23:45,651 [1368620625] Warning : [poller-1] The worker 3 goes down unexpectedly!

I will continue to report on my progress. Thanks

Thread: SnmpBooster, genDevConfig Pending STATE

Thread Tools

Search Thread

Display

SnmpBooster, genDevConfig Pending STATE

Re: SnmpBooster, genDevConfig Pending STATE

Re: SnmpBooster, genDevConfig Pending STATE

Re: SnmpBooster, genDevConfig Pending STATE

Re: SnmpBooster, genDevConfig Pending STATE

Re: SnmpBooster, genDevConfig Pending STATE

Re: SnmpBooster, genDevConfig Pending STATE

Re: SnmpBooster, genDevConfig Pending STATE

Re: SnmpBooster, genDevConfig Pending STATE

Re: SnmpBooster, genDevConfig Pending STATE

Posting Permissions