Results 1 to 6 of 6

Thread: Scheduler doesn't seem to feed the livestatus

  1. #1
    Junior Member
    Join Date
    Aug 2013
    Posts
    3

    Scheduler doesn't seem to feed the livestatus

    Hello,
    we have a realm named "indiric" including a scheduler, a broker and 2 pollers. For one of these pollers, it's a passive poller (in DMZ). Hosts monitored by the poller in DMZ are tagged with "zn_dmzintdiric" tag.
    The poller daemon is tagged too with this.
    The port 7771 was opened between the machine hosting the arbiter (conf flow) and that hosting the scheduler (monitoring flow). With tcpdump, we see exchange between scheduler and passive poller. At the firewall, no rejection appears.
    By cons, when we look at the content of the livestatus live.py script, we find that only the tagged equipment are not filled (Gaston and shindiroint-p) for services.
    When we look at the content of the livestatus (with live.py script), we find that only the tagged equipment are not filled (Gaston-p and shindiroint).
    [['user = 0.05 = 0.05% system% iowait% idle = 0.30 = 99.60%', '80 ', '90', 'boas'], ['',' Gaston-p '] [' user = 12.53 = 38.18% system% iowait% idle = 0.10 = 49.20% ', '80', '90 ',' woodpecker '], [' user = 0.10 = 0.05% system% iowait% idle = 0.60 = 99.25% ', '80 ', '90', 'sar'], ['', 'shindmzint'], ['user = 2.62 = 0.60% system% iowait% idle = 0.00 = 96.78%', '80 ', '90', 'shingraphint ']]
    In the WebUI, services are pending and not host status:
    gaston-p UP 2d 1h TCP OK - 0.000 second response time on port 5666
    Cpu PENDING N / A
    FS_ / PENDING N / A
    FS_ / boot PENDING N / A
    ...
    Someone would have an idea?
    Thanks

  2. #2
    Shinken project leader
    Join Date
    May 2011
    Location
    Bordeaux (France)
    Posts
    2,131

    Re: Scheduler doesn't seem to feed the livestatus

    Look at schedulers and poller debug logs.
    No direct support by personal message. Please open a thread so everyone can see the solution

  3. #3
    Junior Member
    Join Date
    Aug 2013
    Posts
    3

    Re: Scheduler doesn't seem to feed the livestatus

    Hi,

    I did nothing special in the log files.

    I attached the full log.

    The poller one:

    Code:
    2013-10-07 08:43:40,773 [1381135420] Info :  Using the local log file '/var/log/shinken/pollerd.log'
    2013-10-07 08:43:40,798 [1381135420] Warning : Printing stored debug messages prior to our daemonization
    2013-10-07 08:43:40,799 [1381135420] Debug :  modulemanager file /opt/shinken/shinken/modulesmanager.pyc
    2013-10-07 08:43:40,799 [1381135420] Debug :  modulemanager absolute file /opt/shinken/shinken/modulesmanager.pyc
    2013-10-07 08:43:40,799 [1381135420] Debug :  Using modules path: /opt/shinken/shinken/modules
    2013-10-07 08:43:40,799 [1381135420] Debug :  Successfully changed to workdir: /var/run/shinken
    2013-10-07 08:43:40,800 [1381135420] Debug :  Opening pid file: /var/run/shinken/pollerd.pid
    2013-10-07 08:43:40,801 [1381135420] Debug :  Redirecting stdout and stderr as necessary..
    2013-10-07 08:43:40,801 [1381135420] Debug :  We are now fully daemonized :) pid=30846
    2013-10-07 08:43:40,824 [1381135420] Info :  Waiting for initial configuration
    2013-10-07 08:43:59,492 [1381135439] Debug :  The arbiter asked me what I manage. It's {}
    2013-10-07 08:43:59,549 [1381135439] Debug :  [poller] Sending us a configuration {'arbiters': {}, 'global': {'poller_name': u'poller-dmzintdiric', 'max_workers': 0, 'poller_tags': [u'zn_dmzintdiric'], 'modules': [<shinken.objects.module.Module object at 0x7f2fd8e9a9d0>], 'manage_arbiters': False, 'passive': True, 'use_timezone': u'FR/Paris', 'satellitemap': {}, 'polling_interval': 1, 'max_plugins_output_length': 65536, 'min_workers': 0, 'processes_by_worker': 256}, 'schedulers': {1: {'name': u'scheduler-intdiric', 'instance_id': 0, 'address': u'shingraphint.matrics.fr', 'active': True, 'push_flavor': 692333, 'port': 7768}}}
    2013-10-07 08:43:59,549 [1381135439] Info :  [poller-dmzintdiric] Passive mode enabled.
    2013-10-07 08:43:59,550 [1381135439] Info :  [poller-dmzintdiric] Using max workers: 2
    2013-10-07 08:43:59,550 [1381135439] Info :  [poller-dmzintdiric] Using min workers: 2
    2013-10-07 08:43:59,551 [1381135439] Info :  [poller-dmzintdiric] Setting our timezone to FR/Paris
    2013-10-07 08:43:59,551 [1381135439] Info :  We have our schedulers: {1: {'wait_homerun': {}, 'name': u'scheduler-intdiric', 'uri': u'PYRO:Checks@shingraphint.matrics.fr:7768', 'actions': {}, 'instance_id': 0, 'running_id': 0, 'address': u'shingraphint.matrics.fr', 'active': True, 'push_flavor': 692333, 'port': 7768}}
    2013-10-07 08:43:59,552 [1381135439] Debug :  Add module object {'configuration_errors': [], 'use': '', 'hash': '', 'name': '', 'tags': set([]), 'modules': [], 'customs': {}, 'configuration_warnings': [], 'module_name': u'NrpeBooster', 'plus': {}, 'module_type': u'nrpe_poller', 'id': 3, 'imported_from': u'/opt/shinken/etc/modules/booster_nrpe.cfg'}
    2013-10-07 08:43:59,552 [1381135439] Info :  [poller-dmzintdiric] Got module: nrpe_poller
    2013-10-07 08:43:59,557 [1381135439] Warning : Importing module logstore_mongodb: No module named pymongo
    2013-10-07 08:43:59,735 [1381135439] Info :  [NRPEPoller] Get a nrpe poller module for plugin NrpeBooster
    2013-10-07 08:43:59,735 [1381135439] Info :  Trying to init module: NrpeBooster
    2013-10-07 08:43:59,736 [1381135439] Info :  [NRPEPoller] Initialization of the nrpe poller module
    2013-10-07 08:43:59,736 [1381135439] Info :  I correctly loaded the modules: [NrpeBooster]
    2013-10-07 08:43:59,740 [1381135439] Info :  [poller-dmzintdiric] Allocating new fork Worker: 0
    2013-10-07 08:43:59,746 [1381135439] Info :  [poller-dmzintdiric] Allocating new nrpe_poller Worker: 1
    2013-10-07 08:43:59,748 [1381135439] Debug :  Loop turn
    2013-10-07 08:43:59,752 [1381135439] Info :  [NRPEPoller] Module started!
    2013-10-07 08:44:00,749 [1381135440] Debug :  ========================
    2013-10-07 08:44:00,752 [1381135440] Debug :  [1][scheduler-intdiric][fork] Stats: Workers:0 (Queued:0 TotalReturnWait:0)
    2013-10-07 08:44:00,752 [1381135440] Debug :  [1][scheduler-intdiric][nrpe_poller] Stats: Workers:1 (Queued:0 TotalReturnWait:0)
    2013-10-07 08:44:00,753 [1381135440] Debug :  Wait ratio: 1.000000
    2013-10-07 08:44:00,753 [1381135440] Debug :  Loop turn
    The scheduler one:

    Code:
    2013-10-07 08:43:28,349 [1381135408] Info :  Using the local log file '/opt/shinken/var/log/shinken/schedulerd.log'
    2013-10-07 08:43:28,373 [1381135408] Warning : Printing stored debug messages prior to our daemonization
    2013-10-07 08:43:28,374 [1381135408] Debug :  modulemanager file /opt/shinken/shinken/modulesmanager.pyc
    2013-10-07 08:43:28,374 [1381135408] Debug :  modulemanager absolute file /opt/shinken/shinken/modulesmanager.pyc
    2013-10-07 08:43:28,374 [1381135408] Debug :  Using modules path: /opt/shinken/shinken/modules
    2013-10-07 08:43:28,374 [1381135408] Debug :  Successfully changed to workdir: /opt/shinken/var/run/shinken
    2013-10-07 08:43:28,375 [1381135408] Debug :  Opening pid file: /opt/shinken/var/run/shinken/schedulerd.pid
    2013-10-07 08:43:28,375 [1381135408] Debug :  Redirecting stdout and stderr as necessary..
    2013-10-07 08:43:28,375 [1381135408] Debug :  We are now fully daemonized :) pid=15051
    2013-10-07 08:43:28,448 [1381135408] Info :  [scheduler] General interface is at: PYRO:ForArbiter@0.0.0.0:7768
    2013-10-07 08:43:28,449 [1381135408] Info :  Waiting for initial configuration
    2013-10-07 08:43:59,420 [1381135439] Debug :  The arbiter asked me what I manage. It's {}
    2013-10-07 08:43:59,542 [1381135439] Info :  New configuration received
    2013-10-07 08:43:59,795 [1381135439] Debug :  Conf received at 1381135439. Unserialized in 0 secs
    2013-10-07 08:43:59,795 [1381135439] Debug :  Setting our timezone to FR/Paris
    2013-10-07 08:43:59,795 [1381135439] Debug :  I've got [<shinken.objects.module.Module object at 0x2c480d0>] modules
    2013-10-07 08:44:00,930 [1381135440] Debug :  Get a Mongodb retention scheduler module for plugin MongodbRetention
    2013-10-07 08:44:00,931 [1381135440] Info :  Trying to init module: MongodbRetention
    2013-10-07 08:44:00,931 [1381135440] Debug :  Initialization of the mongodb module
    2013-10-07 08:44:01,029 [1381135441] Info :  I correctly loaded the modules: [MongodbRetention]
    2013-10-07 08:44:01,030 [1381135441] Debug :  The Checks Interface uri is: PYRO:Checks@0.0.0.0:7768
    2013-10-07 08:44:01,030 [1381135441] Debug :  The Broks Interface uri is: PYRO:Broks@0.0.0.0:7768
    2013-10-07 08:44:01,030 [1381135441] Info :  Loading configuration.
    2013-10-07 08:44:01,050 [1381135441] Debug :  Late command relink in 0
    2013-10-07 08:44:01,050 [1381135441] Debug :  Changing the tick to 3600 for the function update_retention_file
    2013-10-07 08:44:01,050 [1381135441] Debug :  Changing the tick to 900 for the function clean_queues
    2013-10-07 08:44:01,051 [1381135441] Info :  New configuration loaded
    2013-10-07 08:44:01,051 [1381135441] Debug :  hook_point: MongodbRetention: True load_retention
    2013-10-07 08:44:01,051 [1381135441] Debug :  MongodbRetention] asking me to load the retention objects
    2013-10-07 08:44:01,198 [1381135441] Info :  [MongodbRetention] Retention objects loaded successfully.
    2013-10-07 08:44:01,198 [1381135441] Debug :  hook_point: MongodbRetention: False pre_scheduler_mod_start
    2013-10-07 08:44:01,199 [1381135441] Info :  [scheduler-intdiric] First scheduling launched
    2013-10-07 08:44:01,251 [1381135441] Info :  [scheduler-intdiric] First scheduling done
    2013-10-07 08:44:01,252 [1381135441] Debug :  Init connection with PYRO:Schedulers@shindmzint.matrics.fr:7771
    2013-10-07 08:44:01,256 [1381135441] Info :  Connection OK to the poller poller-dmzintdiric
    2013-10-07 08:44:01,257 [1381135441] Debug :  First loop at 1381135441
    2013-10-07 08:44:01,262 [1381135441] Info :  A new broker just connected : broker-intdiric
    2013-10-07 08:44:01,263 [1381135441] Debug :  Skipping initial broks? False
    2013-10-07 08:44:01,331 [1381135441] Info :  [scheduler-intdiric] Created 116 initial Broks for broker broker-intdiric
    2013-10-07 08:44:01,338 [1381135441] Info :  A new broker just connected : broker-master
    2013-10-07 08:44:01,338 [1381135441] Debug :  Skipping initial broks? False
    2013-10-07 08:44:01,402 [1381135441] Info :  [scheduler-intdiric] Created 116 initial Broks for broker broker-master
    2013-10-07 08:44:02,258 [1381135442] Debug :  Load: (sleep) 31.83 (average: 1.00) -> 0%
    2013-10-07 08:44:02,258 [1381135442] Debug :  hook_point: MongodbRetention: False get_new_actions
    2013-10-07 08:44:02,264 [1381135442] Debug :  Time to send 0 broks (after 0 secs)
    2013-10-07 08:44:02,265 [1381135442] Debug :  I will send actions to the poller {'passive': True, 'last_connection': 1381135441.252135, 'name': u'poller-dmzintdiric', 'poller_tags': [u'zn_dmzintdiric'], 'uri': u'PYRO:Schedulers@shindmzint.matrics.fr:7771', 'instance_id': 2, 'reactionner_tags': [], 'address': u'shindmzint.matrics.fr', 'active': True, 'port': 7771, 'con': <Pyro4.core.Proxy at 0x2d174d0, connected, for PYRO:Schedulers@shindmzint.matrics.fr:7771>}
    2013-10-07 08:44:02,265 [1381135442] Debug :  Sending 0 actions
    2013-10-07 08:44:02,267 [1381135442] Debug :  I will get actions from the poller {'passive': True, 'last_connection': 1381135441.252135, 'name': u'poller-dmzintdiric', 'poller_tags': [u'zn_dmzintdiric'], 'uri': u'PYRO:Schedulers@shindmzint.matrics.fr:7771', 'instance_id': 2, 'reactionner_tags': [], 'address': u'shindmzint.matrics.fr', 'active': True, 'port': 7771, 'con': <Pyro4.core.Proxy at 0x2d174d0, connected, for PYRO:Schedulers@shindmzint.matrics.fr:7771>}
    2013-10-07 08:44:02,269 [1381135442] Debug :  Received 0 passive results
    2013-10-07 08:44:02,269 [1381135442] Debug :  Checks: total 56, scheduled 56, inpoller 0, zombies 0, notifications 1
    2013-10-07 08:44:02,269 [1381135442] Debug :  Latency (avg/min/max): 0.61/0.00/1.80
    2013-10-07 08:44:02,269 [1381135442] Debug :  Nb Broks send: 242
    2013-10-07 08:44:02,270 [1381135442] Debug :  Check average = 0 checks/s
    2013-10-07 08:44:03,271 [1381135443] Debug :  Load: (sleep) 1.00 (average: 1.00) -> 0%
    2013-10-07 08:44:03,271 [1381135443] Debug :  hook_point: MongodbRetention: False get_new_actions
    2013-10-07 08:44:03,272 [1381135443] Debug :  Time to send 0 broks (after 0 secs)
    2013-10-07 08:44:03,275 [1381135443] Debug :  I will send actions to the poller {'passive': True, 'last_connection': 1381135441.252135, 'name': u'poller-dmzintdiric', 'poller_tags': [u'zn_dmzintdiric'], 'uri': u'PYRO:Schedulers@shindmzint.matrics.fr:7771', 'instance_id': 2, 'reactionner_tags': [], 'address': u'shindmzint.matrics.fr', 'active': True, 'port': 7771, 'con': <Pyro4.core.Proxy at 0x2d174d0, connected, for PYRO:Schedulers@shindmzint.matrics.fr:7771>}
    2013-10-07 08:44:03,275 [1381135443] Debug :  Sending 0 actions
    2013-10-07 08:44:03,277 [1381135443] Debug :  I will get actions from the poller {'passive': True, 'last_connection': 1381135441.252135, 'name': u'poller-dmzintdiric', 'poller_tags': [u'zn_dmzintdiric'], 'uri': u'PYRO:Schedulers@shindmzint.matrics.fr:7771', 'instance_id': 2, 'reactionner_tags': [], 'address': u'shindmzint.matrics.fr', 'active': True, 'port': 7771, 'con': <Pyro4.core.Proxy at 0x2d174d0, connected, for PYRO:Schedulers@shindmzint.matrics.fr:7771>}
    Any ideas to troubleshoot further ?

    Regards

  4. #4
    Shinken project leader
    Join Date
    May 2011
    Location
    Bordeaux (France)
    Posts
    2,131

    Re: Scheduler doesn't seem to feed the livestatus

    The connexion and the poller definition for the passive seems good. Hunt on the tag part to see why checks are not sent to this poller.

    Ps: don't use nrpebooster unless you really know what it means to use it for nrpe versions.
    No direct support by personal message. Please open a thread so everyone can see the solution

  5. #5
    Junior Member
    Join Date
    Aug 2013
    Posts
    3

    Re: Scheduler doesn't seem to feed the livestatus

    Hi,
    we have NRPE 2.12 and NRPE 2.14. NRPE is required by the customer.
    With a tcpdump and run a manually check_nrpe with and without the '-n' option, I have the result:

    [matrics@shindmzint libexec]$ ./check_nrpe -H gaston-p
    NRPE v2.14
    [matrics@shindmzint libexec]$ ./check_nrpe -H gaston-p -n
    CHECK_NRPE: Error receiving data from daemon.
    [matrics@shindmzint libexec]$

    In the /var/log/messages, I have
    Oct 10 09:40:23 gaston xinetd[2290]: START: nrpe pid=20813 from=172.31.2.181
    Oct 10 09:40:23 gaston nrpe[20813]: INFO: SSL/TLS initialized. All network traffic will be encrypted.
    Oct 10 09:40:23 gaston xinetd[2290]: EXIT: nrpe status=0 pid=20813 duration=0(sec)

    Oct 10 09:40:26 gaston xinetd[2290]: START: nrpe pid=20814 from=172.31.2.181
    Oct 10 09:40:26 gaston nrpe[20814]: INFO: SSL/TLS initialized. All network traffic will be encrypted.
    Oct 10 09:40:26 gaston nrpe[20814]: Error: Could not complete SSL handshake. 1
    Oct 10 09:40:26 gaston xinetd[2290]: EXIT: nrpe status=0 pid=20814 duration=0(sec)

    For this target, I have a permanent error "Error: Could not complete SSL handshake. 1".

    For another target, with the same NRPE version, I have no error.
    For this, the communication is only for all targets via xinetd.

    Is it possible to simulate or trace specific information for the NrpeBooster from the poller?

    Thanks for your ideas

    PS: for the tag, it's ok
    RePS: we try for this without NrpeBooster and it's work normaly, but it's just a workaround. The NrpeBooster really great interest

  6. #6
    Shinken project leader
    Join Date
    May 2011
    Location
    Bordeaux (France)
    Posts
    2,131

    Re: Scheduler doesn't seem to feed the livestatus

    It's not possible so easily, that's why booster is hard to use and setup.
    No direct support by personal message. Please open a thread so everyone can see the solution

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •