Page 2 of 2 FirstFirst 12
Results 11 to 19 of 19

Thread: SnmpBooster, genDevConfig Pending STATE

  1. #11
    Administrator
    Join Date
    Dec 2011
    Posts
    278

    Re: SnmpBooster, genDevConfig Pending STATE

    In order, as described in the wiki:

    Arbiter registers all objects in memcached
    Scheduler looks up the objects in memcaches
    Poller looks up the objects in memcached and does collection
    Poller updates the objects in memcached and returns check results

    Each daemon always calls the same snmp_poller.py file. So you can debug everything in there. You can also start the arbiter manually using :
    cd /usr/local/shinken
    ./bin/shinken-arbiter

    You can then scroll to see if you have any errors and where they happen.

    Did you find what what file was causing the file error in your configuration?

    cheers,

    xkilian

  2. #12
    Junior Member
    Join Date
    May 2013
    Posts
    10

    Re: SnmpBooster, genDevConfig Pending STATE

    The IOError are gone.

    I found two lines in my snmp_poller.py causing the "Error [poller-1] The worker 1 goes down unexpectedly!"
    when I comment theses two lines SNMP request get send and the worker is not going down.

    Line 808 "snmpoid.raw_old_value = snmpoid.raw_value"
    Code:
        # Save old datas
        #for oid in self.obj.frequences[self.check_interval].services[self.serv_key].oids.values():
        for service in self.obj.frequences[self.check_interval].services.values():
          for snmpoid in service.oids.values():
            snmpoid.old_value = snmpoid.value
            snmpoid.raw_old_value = snmpoid.raw_value
    Line 878-879 "v2c.apiBulkPDU.setVarBinds(self.reqPDU,
    [(x, v2c.null) for x in self.headVars])"

    Code:
        # Build PDU
        self.reqPDU = v2c.GetBulkRequestPDU()
        v2c.apiBulkPDU.setDefaults(self.reqPDU)
        v2c.apiBulkPDU.setNonRepeaters(self.reqPDU, 0)
        v2c.apiBulkPDU.setMaxRepetitions(self.reqPDU, self.max_repetitions)
        v2c.apiBulkPDU.setVarBinds(self.reqPDU,
                      [(x, v2c.null) for x in self.headVars])
    I used this command to get the file
    Are you using a different file ?

    with the lines commented out the status is now UNKNOWN with the message
    Code:
    SNMP Request error: 'module' object has no attribute 'null'

  3. #13
    Administrator
    Join Date
    Dec 2011
    Posts
    278

    Re: SnmpBooster, genDevConfig Pending STATE

    Hmmm.. You are going down the wrong path. This problem is much much simpler.

    snmp_poller.py must be missing access to a file or not loading something, hence to file error. Do you have the traceback of the file error?! Can you paste it.

    My feeling is that snmp_poller.py cannot read your *.ini files in /usr/local/shinken/etc/packs/network/SnmpBooster/
    Put debugging messages around where it loads the ini files and what files it is actualy is loading.


    When Shinken is started you can query your memcached to see if the snmp_poller.py attached to the Arbiter actually created the host and services in memcached.

    You can also add a few debug statements in snmp_poller.py just to output all the keys prepared from the ini files. (Or just seeing if it actually found any!)

  4. #14
    Junior Member
    Join Date
    May 2013
    Posts
    10

    Re: SnmpBooster, genDevConfig Pending STATE

    I can't find who is trying to aces the mystery file.

    sudo grep IOError -b5 -a5 *.log
    Code:
    arbiter.log-0-2013-05-17 11:50:13,732 [1368805813] Warning : Printing stored debug messages prior to our daemonization
    arbiter.log:105:2013-05-17 11:50:13,741 [1368805813] Warning : IOError(2, 'No such file or directory')
    --
    nagios.log-2781-[1368805813] Info :  [broker-1] We have our schedulers: {0: {'broks': {}, 'last_connection': 0, 'name': u'scheduler-1', 'uri': u'PYROLOC://localhost:7768/Broks', 'instance_id': 0, 'running_id': 0, 'address': u'localhost', 'active': True, 'push_flavor': 944448, 'port': 7768}} 
    nagios.log-3060-[1368805813] Info :  New configuration loaded
    nagios.log-3108-[1368805813] Info :  [poller-1] Connection OK with scheduler scheduler-1
    nagios.log-3183-[1368805813] Info :  [reactionner-1] Connection OK with scheduler scheduler-1
    nagios.log-3263-[1368805813] Info :  [broker-1] We have our arbiters: {0: {'broks': {}, 'last_connection': 0, 'name': u'Arbiter-Master', 'uri': u'PYROLOC://localhost:7770/Broks', 'instance_id': 0, 'running_id': 0, 'address': u'localhost', 'port': 7770}} 
    nagios.log:3504:[1368805813] Warning : IOError(2, 'No such file or directory')
    nagios.log-3567-[1368805813] Info :  [poller-1] Using max workers: 2
    nagios.log-3622-[1368805813] Info :  [reactionner-1] Using max workers: 15
    nagios.log-3683-[1368805813] Info :  [broker-1] We have our pollers: {0: {'broks': {}, 'passive': False, 'last_connection': 0, 'name': u'poller-1', 'poller_tags': ['None'], 'uri': u'PYROLOC://localhost:7771/Broks', 'instance_id': 0, 'reactionner_tags': [], 'running_id': 0, 'address': u'localhost', 'active': True, 'port': 7771}}
    nagios.log-3999-[1368805813] Info :  [poller-1] Using min workers: 2
    nagios.log-4054-[1368805813] Info :  [reactionner-1] Using min workers: 1
    --
    nagios.log-56565-[1368805813] Info :  Configuration Loaded
    nagios.log-56609-[1368805813] Info :  Trying to initialize additional groups for the daemon
    nagios.log-56686-[1368805813] Warning : Stale pidfile exists (no or invalid or unreadable content). Reusing it.
    nagios.log-56781-[1368805813] Info :  Initializing Pyro connection with host:localhost port:7770 ssl:False
    nagios.log-56873-[1368805813] Warning : Printing stored debug messages prior to our daemonization
    nagios.log:56954:[1368805813] Warning : IOError(2, 'No such file or directory')
    nagios.log-57017-[1368805823] SERVICE ALERT: 192.168.254.253;if.FastEthernet0_20;UNKNOWN;SOFT;1;FROM CACHE: FastEthernet0_20: Instance mapping not found. Please check your config
    nagios.log-57179-[1368805824] SERVICE ALERT: 192.168.254.253;if.GigabitEthernet0_1;UNKNOWN;SOFT;1;FROM CACHE: GigabitEthernet0_1: Instance mapping not found. Please check your config
    nagios.log-57345-[1368805824] Info :  [SnmpBooster] Forced check for this host/service: 192.168.254.253/if.FastEthernet0_20
    nagios.log-57454-[1368805825] Info :  [SnmpBooster] Forced check for this host/service: 192.168.254.253/if.GigabitEthernet0_1
    nagios.log-57565-[1368805826] Info :  [SnmpBooster] Forced check for this host/service: 192.168.254.253/chassis
    --
    schedulerd.log-61229-2013-05-17 11:50:13,874 [1368805813] Debug :  Changing the tick to 3600 for the function update_retention_file
    schedulerd.log-61341-2013-05-17 11:50:13,874 [1368805813] Debug :  Changing the tick to 900 for the function clean_queues
    schedulerd.log-61443-2013-05-17 11:50:13,874 [1368805813] Info :  New configuration loaded
    schedulerd.log-61515-2013-05-17 11:50:13,874 [1368805813] Debug :  hook_point: PickleRetention: True load_retention
    schedulerd.log-61611-2013-05-17 11:50:13,874 [1368805813] Debug :  [PickleRetentionGeneric]Reading from retention_file /tmp/retention.dat
    schedulerd.log:61729:2013-05-17 11:50:13,874 [1368805813] Warning : IOError(2, 'No such file or directory')
    schedulerd.log-61816-2013-05-17 11:50:13,875 [1368805813] Debug :  hook_point: SnmpBooster: False load_retention
    schedulerd.log-61909-2013-05-17 11:50:13,875 [1368805813] Debug :  hook_point: PickleRetention: False pre_scheduler_mod_start
    schedulerd.log-62015-2013-05-17 11:50:13,875 [1368805813] Debug :  hook_point: SnmpBooster: False pre_scheduler_mod_start
    schedulerd.log-62117-2013-05-17 11:50:13,875 [1368805813] Debug :  Starting initial broks
    schedulerd.log-62187-2013-05-17 11:50:13,875 [1368805813] Debug :  Skipping initial broks? False
    I will keep looking.

  5. #15
    Junior Member
    Join Date
    May 2013
    Posts
    10

    Re: SnmpBooster, genDevConfig Pending STATE

    Ok I am trying somthing else. lets go with the minimum.

    shinken-specific.cfg
    Code:
    define module {
     modules Apache_passwd,ActiveDir_UI,Cfg_password,PNP_UI
     manage_acl 1
     play_sound 0
     host 0.0.0.0
     module_type webui
     allow_html_output 0
     max_output_length 100
     module_name WebUI
     auth_secret CHANGE_ME
     port 7767
    }
    
    define module {
     username user
     basedn DC=google,DC=com
     module_type ad_webui
     mode ad
     module_name ActiveDir_UI
     password password
    }
    
    define module {
     passwd /usr/local/shinken/etc/htpasswd.users
     module_name Apache_passwd
     module_type passwd_webui
    }
    
    define module {
     module_name PNP_UI
     module_type pnp_webui
     uri http://172.22.254.62/pnp4nagios
    }
    
    }
    
    define module {
     archive_path archives/
     module_name Simple-log
     module_type simple_log
     path nagios.log
    }
    
    define module {
     module_name PickleRetention
     module_type pickle_retention_file_generic
     path /tmp/retention.dat
    }
    
    define module {
     command_file rw/nagios.cmd
     module_name CommandFile
     module_type named_pipe
    }
    
    define module {
     module_name Cfg_password
     module_type cfg_password_webui
    }
    
    define poller {
     poller_name poller-1
     data_timeout 120
     check_interval 60
     polling_interval 1
     modules SnmpBooster
     processes_by_worker 256
     manage_sub_realms 0
     max_workers 0
     timeout 3
     address localhost
     realm All
     max_check_attempts 3
     min_workers 0
     port 7771
    }
    
    define realm {
     default 1
     realm_name All
    }
    
    define reactionner {
     data_timeout 120
     check_interval 60
     polling_interval 1
     modules 
     reactionner_name reactionner-1
     manage_sub_realms 0
     max_workers 15
     spare 0
     timeout 3
     address localhost
     realm All
     max_check_attempts 3
     min_workers 1
     port 7769
    }
    
    define broker {
     broker_name broker-1
     data_timeout 120
     check_interval 60
     modules Livestatus, Simple-log, WebUI,NPCDMOD
     port 7772
     manage_sub_realms 1
     spare 0
     timeout 3
     address localhost
     realm All
     max_check_attempts 3
     manage_arbiters 1
    }
    
    define arbiter {
     modules CommandFile,SnmpBooster
     spare 0
     address localhost
     port 7770
     arbiter_name Arbiter-Master
    }
    
    define scheduler {
     data_timeout 120
     check_interval 60
     weight 1
     skip_initial_broks 0
     modules ,PickleRetention,SnmpBooster
     spare 0
     timeout 3
     address localhost
     scheduler_name scheduler-1
     max_check_attempts 3
     realm All
     port 7768
    }
    
    define receiver {
     data_timeout 120
     check_interval 60
     receiver_name receiver-1
     direct_routing 0
     modules 
     spare 0
     timeout 3
     address localhost
     realm All
     max_check_attempts 3
     port 7773
    }
    
    define module {
    
       module_name     SnmpBooster
       module_type     snmp_poller
       datasource      /usr/local/shinken/etc/packs/network/SnmpBooster/  
       memcached_host    127.0.0.1
       memcached_port    11211
    }
    templates.cfg
    Code:
    define service {
    	name        default-snmp-template
        check_command	 check_snmp_booster!$_SERVICEDSTEMPLATE$!$_SERVICEINST$!$_SERVICETRIGGERGROUP$
        _inst          None
    	_triggergroup      None
        max_check_attempts   3
    	check_interval     1
    	retry_interval     1
        use           generic-host
    	register        0
    }
    
    define host{
        name          SnmpBooster-host
        alias          SnmpBooster-Host-Template
        check_command		check_host_alive
        max_check_attempts   3
    	check_interval     1
    	retry_interval     1
    	use           generic-host
    	register        0
    }
    
    
    define command {
      command_name  check_snmp_booster
      command_line  check_snmp_booster -H $HOSTNAME$ -C $SNMPCOMMUNITYREAD$ -V 2c -t $ARG1$ -i $_SERVICEINST$ -T $_SERVICETRIGGERGROUP$
      module_type   snmp_poller ; This is required to tell teh poller how to execute the check
    }
    SnmpBooster.ini
    Code:
    [DATASOURCE]
    
     OidsysUptime = .1.3.6.1.2.1.1.3.0
     [sysUptime]
       ds_type = COUNTER
       ds_oid = OidsysUptime
    [DSTEMPLATE]
    
     [UpTime]
       ds = sysUptime
    host.cfg
    Code:
    define host {
    	host_name		192.168.254.253
    	display_name		192.168.254.253
    	_sys_location		EBBA
    	address			192.168.254.253
    	register		1
    	use			SnmpBooster-host
    }
    
    define service {
    
    	host_name		192.168.254.253
    	service_description	SysUptime
    	display_name		System Up Time
    	_dstemplate		UpTime
    	register		1
    	use			default-snmp-template
    }
    I am missing something because the poller is complaining
    Code:
    Error :  [SnmpBooster] No frequences found for this key: ('UpTime', None, None)

  6. #16
    Administrator
    Join Date
    Dec 2011
    Posts
    278

    Re: SnmpBooster, genDevConfig Pending STATE

    Good idea to simplify.

    From what I see, the arbiter is not writing to memcached and probably the scheduler as well.

    If you follow the steps, I put in another SnmpBooster related thread, you can dump keys and values from your memcached after the arbiter has read your configuration.

    I still have the feeling you are missing a dependency somewhere related to memcached.. Look are what snmp_booster imports at the top of the script and make sure they are all there. You can test the imports from the python command line. But I would expect you would have a trace back or an error message if that was the case.

    # python
    import xxx
    import yyy

    hmm.. Curious problem,

  7. #17
    Junior Member
    Join Date
    May 2013
    Posts
    10

    Re: SnmpBooster, genDevConfig Pending STATE

    I checked the dependency in snmp_poller.py and everything is fine.

    Code:
    >>> import glob
    >>> import signal
    >>> import time
    >>> import socket
    >>> import struct
    >>> import copy
    >>> import binascii
    >>> import binascii
    >>> import getopt
    >>> import binascii
    >>> import shlex
    >>> import operator
    >>> import math
    >>> import datetime
    >>> import Queue
    >>> import memcache
    >>> import configobj
    >>> import pysnmp
    >>> import pyasn1

  8. #18
    Junior Member
    Join Date
    May 2013
    Posts
    10

    Re: SnmpBooster, genDevConfig Pending STATE

    Any idea what i should change to fix the error message ?
    Code:
    Error :  [SnmpBooster] No frequences found for this key: ('UpTime', None, None)

  9. #19
    Administrator
    Join Date
    Dec 2011
    Posts
    278

    Re: SnmpBooster, genDevConfig Pending STATE

    Hello,

    I have to look at the code not sure. But it is related to communication with memcache. It is looking for data that should have been written there by the arbiter and it cannot find it.

    I will send you the link to the ftp site so you can send me the full debug logs of all the daemons.

    Cheers,

    xkilian

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •