Passive checks logging

[marcochapeau]

Hey everyone,

We have a nice Shinken system running that does both active and passive checks. All our passive checks are basically SNMP Traps sent to shinken with submit_check_result (via snmptt). So far so good, we get our alarms in thruk and it makes us very happy... except for one thing: we don't get any "PASSIVE CHECK SERVICE" in nagios.log. We do manage to get some "EXTERNAL COMMAND" log lines if I set log_level to INFO, but this also generates a lot of unuseful logs. I did set log_passive_checks=1 in nagios.cfg, but according to docs it is meant to be the default and even if explicitly enabled it doesn't change anything.

Does anyone have a clue what might be happening or what other info I should post to get some help on this ?

Thanks,
Marc.

[naparuba]

Do you have theses lines with INFO level or not?

[marcochapeau]

Nope, not even when I enable INFO level. I tried debugging the broker, but I couldn't see much (also, not sure what to look for). Maybe the debug log could show me which log classes are really active ?

[marcochapeau]

I'm burning my brains out on this one. I just can't figure out what is wrong.

Nap ? Would you have a clue what could happen ? Any ideas of things I could look at ?

[marcochapeau]

Hi,

I finally found what was wrong. It wasn't just the passive checks. Something we hadn't noticed until yesterday is that the passive check where all coming from 2 of the sub-realms. It was actually the sub-realms who where not being logged by the broker. Strange thing, the graphite broker module was sending data for those machines all along...

The issue was a typo in shinken-specific.cfg that was silently accepted. It's not the first time I see an issue with subrealms and errors in shinken-specific.cfg. It might be worth looking into adding some form of parsing there so that at least the corresponding module can complain.

The big issue here was that we had no errors, and the broker was still working ok for other things in this realm. Only alert logging in simple-log and livestatus was broken.

If you need anymore info, feel free to ask.

[naparuba]

What was the typo, did it make an invalid situation for the broker? (like no modules or things like that?).

"what ever" property name is by design, to allow modules to use the name they want in objects, we can't to a "break" on unknown property name.

[marcochapeau]

My schedulers where all defined as follow:

define scheduler {
 data_timeout 120
 check_interval 60
 weight 1
 skip_initial_broks 0
 modules ,PickleRetention
 spare 0
 timeout 3
 address 10.129.31.150
 scheduler_name scheduler-claranet
 max_check_attempts 3
 port 7768
 realm Claranet
}
define scheduler {
 data_timeout 120
 check_interval 60
 weight 1
 skip_initial_broks 0
 modules ,PickleRetention
 spare 0
 timeout 3
 address 192.168.10.113
 scheduler_name scheduler-frankfurt
 max_check_attempts 3
 port 7768
 realm Frankfurt
}
define scheduler {
 data_timeout 120
 check_interval 60
 weight 1
 skip_initial_broks 0
 modules ,PickleRetention
 spare 0
 timeout 3
 address 192.168.20.113
 scheduler_name scheduler-washington
 max_check_attempts 3
 port 7768
 realm Washington
}

Notice the ',' right before the 'PickleRetention' ? Very perverse, hard to see and not related to a module option. It is basic syntax checking in my opinion. So even if the parser does not complain about unknown keywords, syntax checking should be done.

What do you think ?

[marcochapeau]

Wait, I got that wrong. The arbiter also had the typo. The weird thing is that logging was working just fine for the Claranet realm. Only Washington and Frankfurt were behaving strangely.

define arbiter {
 arbiter_name Arbiter-Master
 modules ,PickleRetentionArbiter
 spare 0
 address 10.129.31.150
 port 7770
 realm All
}
define scheduler {
 data_timeout 120
 check_interval 60
 weight 1
 skip_initial_broks 0
 modules ,PickleRetention
 spare 0
 timeout 3
 address 10.129.31.150
 scheduler_name scheduler-claranet
 max_check_attempts 3
 port 7768
 realm Claranet
}
define scheduler {
 data_timeout 120
 check_interval 60
 weight 1
 skip_initial_broks 0
 modules ,PickleRetention
 spare 0
 timeout 3
 address 192.168.10.113
 scheduler_name scheduler-frankfurt
 max_check_attempts 3
 port 7768
 realm Frankfurt
}
define scheduler {
 data_timeout 120
 check_interval 60
 weight 1
 skip_initial_broks 0
 modules ,PickleRetention
 spare 0
 timeout 3
 address 192.168.20.113
 scheduler_name scheduler-washington
 max_check_attempts 3
 port 7768
 realm Washington
}

[naparuba]

A void object should not be a problem, but maybe the scheduler got such a problem for module linking. can you open a ticket about it on the shinken github? Thanks

[Hudson]

There are many systems that are working for the sake of help to the people. Although there is assignmentexpert so that they can have the familiarity with various options to promote more.