SnmpBooster, genDevConfig Pending STATE

[xkilian]

In order, as described in the wiki:

Arbiter registers all objects in memcached
Scheduler looks up the objects in memcaches
Poller looks up the objects in memcached and does collection
Poller updates the objects in memcached and returns check results

Each daemon always calls the same snmp_poller.py file. So you can debug everything in there. You can also start the arbiter manually using :
cd /usr/local/shinken
./bin/shinken-arbiter

You can then scroll to see if you have any errors and where they happen.

Did you find what what file was causing the file error in your configuration?

cheers,

xkilian

[Miraculor]

The IOError are gone.

I found two lines in my snmp_poller.py causing the "Error [poller-1] The worker 1 goes down unexpectedly!"
when I comment theses two lines SNMP request get send and the worker is not going down.

Line 808 "snmpoid.raw_old_value = snmpoid.raw_value"

    # Save old datas
    #for oid in self.obj.frequences[self.check_interval].services[self.serv_key].oids.values():
    for service in self.obj.frequences[self.check_interval].services.values():
      for snmpoid in service.oids.values():
        snmpoid.old_value = snmpoid.value
        snmpoid.raw_old_value = snmpoid.raw_value

Line 878-879 "v2c.apiBulkPDU.setVarBinds(self.reqPDU,
[(x, v2c.null) for x in self.headVars])"

    # Build PDU
    self.reqPDU = v2c.GetBulkRequestPDU()
    v2c.apiBulkPDU.setDefaults(self.reqPDU)
    v2c.apiBulkPDU.setNonRepeaters(self.reqPDU, 0)
    v2c.apiBulkPDU.setMaxRepetitions(self.reqPDU, self.max_repetitions)
    v2c.apiBulkPDU.setVarBinds(self.reqPDU,
                  [(x, v2c.null) for x in self.headVars])

I used this command to get the file

wget https://raw.github.com/titilambert/s...snmp_poller.py

Are you using a different file ?

with the lines commented out the status is now UNKNOWN with the message

SNMP Request error: 'module' object has no attribute 'null'

[xkilian]

Hmmm.. You are going down the wrong path. This problem is much much simpler.

snmp_poller.py must be missing access to a file or not loading something, hence to file error. Do you have the traceback of the file error?! Can you paste it.

My feeling is that snmp_poller.py cannot read your *.ini files in /usr/local/shinken/etc/packs/network/SnmpBooster/
Put debugging messages around where it loads the ini files and what files it is actualy is loading.


When Shinken is started you can query your memcached to see if the snmp_poller.py attached to the Arbiter actually created the host and services in memcached.

You can also add a few debug statements in snmp_poller.py just to output all the keys prepared from the ini files. (Or just seeing if it actually found any!)

[Miraculor]

I can't find who is trying to aces the mystery file.

sudo grep IOError -b5 -a5 *.log

arbiter.log-0-2013-05-17 11:50:13,732 [1368805813] Warning : Printing stored debug messages prior to our daemonization
arbiter.log:105:2013-05-17 11:50:13,741 [1368805813] Warning : IOError(2, 'No such file or directory')
--
nagios.log-2781-[1368805813] Info :  [broker-1] We have our schedulers: {0: {'broks': {}, 'last_connection': 0, 'name': u'scheduler-1', 'uri': u'PYROLOC://localhost:7768/Broks', 'instance_id': 0, 'running_id': 0, 'address': u'localhost', 'active': True, 'push_flavor': 944448, 'port': 7768}} 
nagios.log-3060-[1368805813] Info :  New configuration loaded
nagios.log-3108-[1368805813] Info :  [poller-1] Connection OK with scheduler scheduler-1
nagios.log-3183-[1368805813] Info :  [reactionner-1] Connection OK with scheduler scheduler-1
nagios.log-3263-[1368805813] Info :  [broker-1] We have our arbiters: {0: {'broks': {}, 'last_connection': 0, 'name': u'Arbiter-Master', 'uri': u'PYROLOC://localhost:7770/Broks', 'instance_id': 0, 'running_id': 0, 'address': u'localhost', 'port': 7770}} 
nagios.log:3504:[1368805813] Warning : IOError(2, 'No such file or directory')
nagios.log-3567-[1368805813] Info :  [poller-1] Using max workers: 2
nagios.log-3622-[1368805813] Info :  [reactionner-1] Using max workers: 15
nagios.log-3683-[1368805813] Info :  [broker-1] We have our pollers: {0: {'broks': {}, 'passive': False, 'last_connection': 0, 'name': u'poller-1', 'poller_tags': ['None'], 'uri': u'PYROLOC://localhost:7771/Broks', 'instance_id': 0, 'reactionner_tags': [], 'running_id': 0, 'address': u'localhost', 'active': True, 'port': 7771}}
nagios.log-3999-[1368805813] Info :  [poller-1] Using min workers: 2
nagios.log-4054-[1368805813] Info :  [reactionner-1] Using min workers: 1
--
nagios.log-56565-[1368805813] Info :  Configuration Loaded
nagios.log-56609-[1368805813] Info :  Trying to initialize additional groups for the daemon
nagios.log-56686-[1368805813] Warning : Stale pidfile exists (no or invalid or unreadable content). Reusing it.
nagios.log-56781-[1368805813] Info :  Initializing Pyro connection with host:localhost port:7770 ssl:False
nagios.log-56873-[1368805813] Warning : Printing stored debug messages prior to our daemonization
nagios.log:56954:[1368805813] Warning : IOError(2, 'No such file or directory')
nagios.log-57017-[1368805823] SERVICE ALERT: 192.168.254.253;if.FastEthernet0_20;UNKNOWN;SOFT;1;FROM CACHE: FastEthernet0_20: Instance mapping not found. Please check your config
nagios.log-57179-[1368805824] SERVICE ALERT: 192.168.254.253;if.GigabitEthernet0_1;UNKNOWN;SOFT;1;FROM CACHE: GigabitEthernet0_1: Instance mapping not found. Please check your config
nagios.log-57345-[1368805824] Info :  [SnmpBooster] Forced check for this host/service: 192.168.254.253/if.FastEthernet0_20
nagios.log-57454-[1368805825] Info :  [SnmpBooster] Forced check for this host/service: 192.168.254.253/if.GigabitEthernet0_1
nagios.log-57565-[1368805826] Info :  [SnmpBooster] Forced check for this host/service: 192.168.254.253/chassis
--
schedulerd.log-61229-2013-05-17 11:50:13,874 [1368805813] Debug :  Changing the tick to 3600 for the function update_retention_file
schedulerd.log-61341-2013-05-17 11:50:13,874 [1368805813] Debug :  Changing the tick to 900 for the function clean_queues
schedulerd.log-61443-2013-05-17 11:50:13,874 [1368805813] Info :  New configuration loaded
schedulerd.log-61515-2013-05-17 11:50:13,874 [1368805813] Debug :  hook_point: PickleRetention: True load_retention
schedulerd.log-61611-2013-05-17 11:50:13,874 [1368805813] Debug :  [PickleRetentionGeneric]Reading from retention_file /tmp/retention.dat
schedulerd.log:61729:2013-05-17 11:50:13,874 [1368805813] Warning : IOError(2, 'No such file or directory')
schedulerd.log-61816-2013-05-17 11:50:13,875 [1368805813] Debug :  hook_point: SnmpBooster: False load_retention
schedulerd.log-61909-2013-05-17 11:50:13,875 [1368805813] Debug :  hook_point: PickleRetention: False pre_scheduler_mod_start
schedulerd.log-62015-2013-05-17 11:50:13,875 [1368805813] Debug :  hook_point: SnmpBooster: False pre_scheduler_mod_start
schedulerd.log-62117-2013-05-17 11:50:13,875 [1368805813] Debug :  Starting initial broks
schedulerd.log-62187-2013-05-17 11:50:13,875 [1368805813] Debug :  Skipping initial broks? False

I will keep looking.

[Miraculor]

Ok I am trying somthing else. lets go with the minimum.

shinken-specific.cfg

define module {
 modules Apache_passwd,ActiveDir_UI,Cfg_password,PNP_UI
 manage_acl 1
 play_sound 0
 host 0.0.0.0
 module_type webui
 allow_html_output 0
 max_output_length 100
 module_name WebUI
 auth_secret CHANGE_ME
 port 7767
}

define module {
 username user
 basedn DC=google,DC=com
 module_type ad_webui
 mode ad
 module_name ActiveDir_UI
 password password
}

define module {
 passwd /usr/local/shinken/etc/htpasswd.users
 module_name Apache_passwd
 module_type passwd_webui
}

define module {
 module_name PNP_UI
 module_type pnp_webui
 uri http://172.22.254.62/pnp4nagios
}

}

define module {
 archive_path archives/
 module_name Simple-log
 module_type simple_log
 path nagios.log
}

define module {
 module_name PickleRetention
 module_type pickle_retention_file_generic
 path /tmp/retention.dat
}

define module {
 command_file rw/nagios.cmd
 module_name CommandFile
 module_type named_pipe
}

define module {
 module_name Cfg_password
 module_type cfg_password_webui
}

define poller {
 poller_name poller-1
 data_timeout 120
 check_interval 60
 polling_interval 1
 modules SnmpBooster
 processes_by_worker 256
 manage_sub_realms 0
 max_workers 0
 timeout 3
 address localhost
 realm All
 max_check_attempts 3
 min_workers 0
 port 7771
}

define realm {
 default 1
 realm_name All
}

define reactionner {
 data_timeout 120
 check_interval 60
 polling_interval 1
 modules 
 reactionner_name reactionner-1
 manage_sub_realms 0
 max_workers 15
 spare 0
 timeout 3
 address localhost
 realm All
 max_check_attempts 3
 min_workers 1
 port 7769
}

define broker {
 broker_name broker-1
 data_timeout 120
 check_interval 60
 modules Livestatus, Simple-log, WebUI,NPCDMOD
 port 7772
 manage_sub_realms 1
 spare 0
 timeout 3
 address localhost
 realm All
 max_check_attempts 3
 manage_arbiters 1
}

define arbiter {
 modules CommandFile,SnmpBooster
 spare 0
 address localhost
 port 7770
 arbiter_name Arbiter-Master
}

define scheduler {
 data_timeout 120
 check_interval 60
 weight 1
 skip_initial_broks 0
 modules ,PickleRetention,SnmpBooster
 spare 0
 timeout 3
 address localhost
 scheduler_name scheduler-1
 max_check_attempts 3
 realm All
 port 7768
}

define receiver {
 data_timeout 120
 check_interval 60
 receiver_name receiver-1
 direct_routing 0
 modules 
 spare 0
 timeout 3
 address localhost
 realm All
 max_check_attempts 3
 port 7773
}

define module {

   module_name     SnmpBooster
   module_type     snmp_poller
   datasource      /usr/local/shinken/etc/packs/network/SnmpBooster/  
   memcached_host    127.0.0.1
   memcached_port    11211
}

templates.cfg

define service {
	name        default-snmp-template
    check_command	 check_snmp_booster!$_SERVICEDSTEMPLATE$!$_SERVICEINST$!$_SERVICETRIGGERGROUP$
    _inst          None
	_triggergroup      None
    max_check_attempts   3
	check_interval     1
	retry_interval     1
    use           generic-host
	register        0
}

define host{
    name          SnmpBooster-host
    alias          SnmpBooster-Host-Template
    check_command		check_host_alive
    max_check_attempts   3
	check_interval     1
	retry_interval     1
	use           generic-host
	register        0
}


define command {
  command_name  check_snmp_booster
  command_line  check_snmp_booster -H $HOSTNAME$ -C $SNMPCOMMUNITYREAD$ -V 2c -t $ARG1$ -i $_SERVICEINST$ -T $_SERVICETRIGGERGROUP$
  module_type   snmp_poller ; This is required to tell teh poller how to execute the check
}

SnmpBooster.ini

[DATASOURCE]

 OidsysUptime = .1.3.6.1.2.1.1.3.0
 [sysUptime]
   ds_type = COUNTER
   ds_oid = OidsysUptime
[DSTEMPLATE]

 [UpTime]
   ds = sysUptime

host.cfg

define host {
	host_name		192.168.254.253
	display_name		192.168.254.253
	_sys_location		EBBA
	address			192.168.254.253
	register		1
	use			SnmpBooster-host
}

define service {

	host_name		192.168.254.253
	service_description	SysUptime
	display_name		System Up Time
	_dstemplate		UpTime
	register		1
	use			default-snmp-template
}

I am missing something because the poller is complaining

Error :  [SnmpBooster] No frequences found for this key: ('UpTime', None, None)

[xkilian]

Good idea to simplify.

From what I see, the arbiter is not writing to memcached and probably the scheduler as well.

If you follow the steps, I put in another SnmpBooster related thread, you can dump keys and values from your memcached after the arbiter has read your configuration.

I still have the feeling you are missing a dependency somewhere related to memcached.. Look are what snmp_booster imports at the top of the script and make sure they are all there. You can test the imports from the python command line. But I would expect you would have a trace back or an error message if that was the case.

# python
import xxx
import yyy

hmm.. Curious problem,

[Miraculor]

I checked the dependency in snmp_poller.py and everything is fine.

>>> import glob
>>> import signal
>>> import time
>>> import socket
>>> import struct
>>> import copy
>>> import binascii
>>> import binascii
>>> import getopt
>>> import binascii
>>> import shlex
>>> import operator
>>> import math
>>> import datetime
>>> import Queue
>>> import memcache
>>> import configobj
>>> import pysnmp
>>> import pyasn1

[Miraculor]

Any idea what i should change to fix the error message ?

Error :  [SnmpBooster] No frequences found for this key: ('UpTime', None, None)

[xkilian]

Hello,

I have to look at the code not sure. But it is related to communication with memcache. It is looking for data that should have been written there by the arbiter and it cannot find it.

I will send you the link to the ftp site so you can send me the full debug logs of all the daemons.

Cheers,

xkilian