Results 1 to 2 of 2

Thread: HA question: lot of redispatch msgs in arbiter log after network outage recover

  1. #1
    Junior Member
    Join Date
    Jan 2014
    Posts
    23

    HA question: lot of redispatch msgs in arbiter log after network outage recover

    Hi all,

    I noticed something - in my opinion - strange in the arbiter log file after a recovery of a network outage.

    Here's my simple test scenario:

    Code:
    Server A:	eth0 -> XXX.XXX.XXX.100
    		eth1 -> YYY.YYY.YYY.50
    		broker-1, receiver-1, reactionner-1, poller-1, scheduler-1 are running
    			
    Server B:	eth0 -> XXX.XXX.XXX.101
    		eth1 -> YYY.YYY.YYY.51
    		arbiter-2, broker-2 (spare), receiver-2 (spare), reactionner-2 (spare), poller-2 (spare), scheduler-2 (spare) are running
    (note that there is no arbiter-1)

    The NICs eth0 point to a company wide network X (tried to mark it with XXX.XXX.XXX). The web interface is reachable via eth0.
    The NICs eth1 point to a private network Y (YYY.YYY.YYY) for administration purposes and monitoring traffic. All shinken components are bound only to this interface.

    To simulate a network outage I shut down eth1 of Server A. The arbiter on Server B noticed that the shinken daemons on Server A were not reachable anymore and distributed the configurations to the spare daemons. That worked fine. The web interface on Server A is still reachable (with outdated information, discussed that in another thread) and it is also available on Server B now with current information.

    After enabling eth1 on Server A again, I see the following messages in the arbiter log (Server B) file:

    Code:
    #> cat var/arbiter.log | grep -v "PYROLOC" | grep -v "Pinging"
    2014-01-14 15:28:57,731 [1389709737] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:28:58,744 [1389709738] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:28:59,752 [1389709739] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:00,759 [1389709740] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:01,770 [1389709741] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:02,781 [1389709742] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:03,791 [1389709743] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:04,800 [1389709744] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:05,810 [1389709745] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:06,818 [1389709746] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:12,791 [1389709752] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:13,800 [1389709753] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:14,813 [1389709754] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:15,824 [1389709755] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:16,836 [1389709756] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:17,855 [1389709757] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:17,856 [1389709757] Info :  I ask receiver-1 to wait a new conf
    2014-01-14 15:29:18,869 [1389709758] Info :  OK, all schedulers configurations are dispatched :)
    2014-01-14 15:29:18,870 [1389709758] Info :  [All] Trying to send configuration to receiver receiver-1
    2014-01-14 15:29:18,871 [1389709758] Info :  [All] Dispatch OK of configuration to receiver receiver-1
    2014-01-14 15:29:18,872 [1389709758] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:18,873 [1389709758] Info :  I ask receiver-1 to wait a new conf
    2014-01-14 15:29:19,883 [1389709759] Info :  OK, all schedulers configurations are dispatched :)
    2014-01-14 15:29:19,883 [1389709759] Info :  [All] Trying to send configuration to receiver receiver-1
    2014-01-14 15:29:19,884 [1389709759] Info :  [All] Dispatch OK of configuration to receiver receiver-1
    2014-01-14 15:29:19,885 [1389709759] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:19,886 [1389709759] Info :  I ask receiver-1 to wait a new conf
    2014-01-14 15:29:20,897 [1389709760] Info :  OK, all schedulers configurations are dispatched :)
    2014-01-14 15:29:20,897 [1389709760] Info :  [All] Trying to send configuration to receiver receiver-1
    2014-01-14 15:29:20,899 [1389709760] Info :  [All] Dispatch OK of configuration to receiver receiver-1
    2014-01-14 15:29:20,899 [1389709760] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:20,900 [1389709760] Info :  I ask receiver-1 to wait a new conf
    2014-01-14 15:29:21,911 [1389709761] Info :  OK, all schedulers configurations are dispatched :)
    2014-01-14 15:29:21,911 [1389709761] Info :  [All] Trying to send configuration to receiver receiver-1
    2014-01-14 15:29:21,912 [1389709761] Info :  [All] Dispatch OK of configuration to receiver receiver-1
    2014-01-14 15:29:21,913 [1389709761] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:21,914 [1389709761] Info :  I ask receiver-1 to wait a new conf
    2014-01-14 15:29:22,925 [1389709762] Info :  OK, all schedulers configurations are dispatched :)
    2014-01-14 15:29:22,925 [1389709762] Info :  [All] Trying to send configuration to receiver receiver-1
    2014-01-14 15:29:22,927 [1389709762] Info :  [All] Dispatch OK of configuration to receiver receiver-1
    2014-01-14 15:29:22,927 [1389709762] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:22,928 [1389709762] Info :  I ask receiver-1 to wait a new conf
    2014-01-14 15:29:23,941 [1389709763] Info :  OK, all schedulers configurations are dispatched :)
    2014-01-14 15:29:23,942 [1389709763] Info :  [All] Trying to send configuration to receiver receiver-1
    2014-01-14 15:29:23,943 [1389709763] Info :  [All] Dispatch OK of configuration to receiver receiver-1
    2014-01-14 15:29:23,943 [1389709763] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:23,945 [1389709763] Info :  I ask receiver-1 to wait a new conf
    2014-01-14 15:29:24,956 [1389709764] Info :  OK, all schedulers configurations are dispatched :)
    2014-01-14 15:29:24,957 [1389709764] Info :  [All] Trying to send configuration to receiver receiver-1
    2014-01-14 15:29:24,958 [1389709764] Info :  [All] Dispatch OK of configuration to receiver receiver-1
    2014-01-14 15:29:24,959 [1389709764] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:24,960 [1389709764] Info :  I ask receiver-1 to wait a new conf
    2014-01-14 15:29:25,972 [1389709765] Info :  OK, all schedulers configurations are dispatched :)
    2014-01-14 15:29:25,972 [1389709765] Info :  [All] Trying to send configuration to receiver receiver-1
    2014-01-14 15:29:25,974 [1389709765] Info :  [All] Dispatch OK of configuration to receiver receiver-1
    2014-01-14 15:29:25,974 [1389709765] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:25,976 [1389709765] Info :  I ask receiver-1 to wait a new conf
    2014-01-14 15:29:26,988 [1389709766] Info :  OK, all schedulers configurations are dispatched :)
    2014-01-14 15:29:26,988 [1389709766] Info :  [All] Trying to send configuration to receiver receiver-1
    2014-01-14 15:29:26,990 [1389709766] Info :  [All] Dispatch OK of configuration to receiver receiver-1
    2014-01-14 15:29:26,990 [1389709766] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:26,991 [1389709766] Info :  I ask receiver-1 to wait a new conf
    2014-01-14 15:29:28,002 [1389709768] Info :  OK, all schedulers configurations are dispatched :)
    2014-01-14 15:29:28,003 [1389709768] Info :  [All] Trying to send configuration to receiver receiver-1
    2014-01-14 15:29:28,004 [1389709768] Info :  [All] Dispatch OK of configuration to receiver receiver-1
    2014-01-14 15:29:28,005 [1389709768] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:28,006 [1389709768] Info :  I ask receiver-1 to wait a new conf
    2014-01-14 15:29:29,017 [1389709769] Info :  OK, all schedulers configurations are dispatched :)
    2014-01-14 15:29:29,017 [1389709769] Info :  [All] Trying to send configuration to receiver receiver-1
    2014-01-14 15:29:29,019 [1389709769] Info :  [All] Dispatch OK of configuration to receiver receiver-1
    2014-01-14 15:29:29,019 [1389709769] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:29,020 [1389709769] Info :  I ask receiver-1 to wait a new conf
    2014-01-14 15:29:30,030 [1389709770] Info :  OK, all schedulers configurations are dispatched :)
    2014-01-14 15:29:30,030 [1389709770] Info :  [All] Trying to send configuration to receiver receiver-1
    2014-01-14 15:29:30,032 [1389709770] Info :  [All] Dispatch OK of configuration to receiver receiver-1
    2014-01-14 15:29:30,033 [1389709770] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:30,034 [1389709770] Info :  I ask receiver-1 to wait a new conf
    2014-01-14 15:29:31,043 [1389709771] Info :  OK, all schedulers configurations are dispatched :)
    2014-01-14 15:29:31,043 [1389709771] Info :  [All] Trying to send configuration to receiver receiver-1
    2014-01-14 15:29:31,045 [1389709771] Info :  [All] Dispatch OK of configuration to receiver receiver-1
    2014-01-14 15:29:31,045 [1389709771] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:31,047 [1389709771] Info :  I ask receiver-1 to wait a new conf
    2014-01-14 15:29:32,057 [1389709772] Info :  OK, all schedulers configurations are dispatched :)
    2014-01-14 15:29:32,057 [1389709772] Info :  [All] Trying to send configuration to receiver receiver-1
    2014-01-14 15:29:32,059 [1389709772] Info :  [All] Dispatch OK of configuration to receiver receiver-1
    2014-01-14 15:29:32,059 [1389709772] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:32,060 [1389709772] Info :  I ask receiver-1 to wait a new conf
    2014-01-14 15:29:33,071 [1389709773] Info :  OK, all schedulers configurations are dispatched :)
    2014-01-14 15:29:33,071 [1389709773] Info :  [All] Trying to send configuration to receiver receiver-1
    2014-01-14 15:29:33,073 [1389709773] Info :  [All] Dispatch OK of configuration to receiver receiver-1
    2014-01-14 15:29:33,073 [1389709773] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:33,074 [1389709773] Info :  I ask receiver-1 to wait a new conf
    2014-01-14 15:29:34,084 [1389709774] Info :  OK, all schedulers configurations are dispatched :)
    2014-01-14 15:29:34,085 [1389709774] Info :  [All] Trying to send configuration to receiver receiver-1
    2014-01-14 15:29:34,087 [1389709774] Info :  [All] Dispatch OK of configuration to receiver receiver-1
    2014-01-14 15:29:34,088 [1389709774] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:34,089 [1389709774] Info :  I ask receiver-1 to wait a new conf
    2014-01-14 15:29:35,100 [1389709775] Info :  OK, all schedulers configurations are dispatched :)
    2014-01-14 15:29:35,100 [1389709775] Info :  [All] Trying to send configuration to receiver receiver-1
    2014-01-14 15:29:35,101 [1389709775] Info :  [All] Dispatch OK of configuration to receiver receiver-1
    2014-01-14 15:29:35,102 [1389709775] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:35,103 [1389709775] Info :  I ask receiver-1 to wait a new conf
    2014-01-14 15:29:36,114 [1389709776] Info :  OK, all schedulers configurations are dispatched :)
    2014-01-14 15:29:36,114 [1389709776] Info :  [All] Trying to send configuration to receiver receiver-1
    2014-01-14 15:29:36,116 [1389709776] Info :  [All] Dispatch OK of configuration to receiver receiver-1
    2014-01-14 15:29:36,116 [1389709776] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:36,117 [1389709776] Info :  I ask receiver-1 to wait a new conf
    ...
    2014-01-14 15:29:47,289 [1389709787] Warning : The element scheduler-1 have a conf and should not have one! I ask it to idle now
    2014-01-14 15:29:47,291 [1389709787] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:47,293 [1389709787] Info :  I ask receiver-1 to wait a new conf
    2014-01-14 15:29:48,306 [1389709788] Info :  OK, all schedulers configurations are dispatched :)
    2014-01-14 15:29:48,307 [1389709788] Info :  [All] Trying to send configuration to receiver receiver-1
    2014-01-14 15:29:48,308 [1389709788] Info :  [All] Dispatch OK of configuration to receiver receiver-1
    2014-01-14 15:29:48,310 [1389709788] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:48,311 [1389709788] Info :  I ask receiver-1 to wait a new conf
    2014-01-14 15:29:49,323 [1389709789] Info :  OK, all schedulers configurations are dispatched :)
    2014-01-14 15:29:49,324 [1389709789] Info :  [All] Trying to send configuration to receiver receiver-1
    2014-01-14 15:29:49,326 [1389709789] Info :  [All] Dispatch OK of configuration to receiver receiver-1
    2014-01-14 15:29:49,327 [1389709789] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:49,329 [1389709789] Info :  I ask receiver-1 to wait a new conf
    2014-01-14 15:29:50,340 [1389709790] Info :  OK, all schedulers configurations are dispatched :)
    2014-01-14 15:29:50,341 [1389709790] Info :  [All] Trying to send configuration to receiver receiver-1
    2014-01-14 15:29:50,342 [1389709790] Info :  [All] Dispatch OK of configuration to receiver receiver-1
    2014-01-14 15:29:50,344 [1389709790] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:50,345 [1389709790] Info :  I ask receiver-1 to wait a new conf
    2014-01-14 15:29:51,356 [1389709791] Info :  OK, all schedulers configurations are dispatched :)
    2014-01-14 15:29:51,357 [1389709791] Info :  [All] Trying to send configuration to receiver receiver-1
    2014-01-14 15:29:51,358 [1389709791] Info :  [All] Dispatch OK of configuration to receiver receiver-1
    2014-01-14 15:29:51,360 [1389709791] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:51,361 [1389709791] Info :  I ask receiver-1 to wait a new conf
    2014-01-14 15:29:52,373 [1389709792] Info :  OK, all schedulers configurations are dispatched :)
    2014-01-14 15:29:52,373 [1389709792] Info :  [All] Trying to send configuration to receiver receiver-1
    2014-01-14 15:29:52,375 [1389709792] Info :  [All] Dispatch OK of configuration to receiver receiver-1
    2014-01-14 15:29:52,377 [1389709792] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:52,378 [1389709792] Info :  I ask receiver-1 to wait a new conf
    2014-01-14 15:29:53,394 [1389709793] Info :  OK, all schedulers configurations are dispatched :)
    2014-01-14 15:29:53,394 [1389709793] Info :  [All] Trying to send configuration to receiver receiver-1
    2014-01-14 15:29:53,396 [1389709793] Info :  [All] Dispatch OK of configuration to receiver receiver-1
    2014-01-14 15:29:53,397 [1389709793] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:53,399 [1389709793] Info :  I ask receiver-1 to wait a new conf
    2014-01-14 15:29:54,410 [1389709794] Info :  OK, all schedulers configurations are dispatched :)
    2014-01-14 15:29:54,410 [1389709794] Info :  [All] Trying to send configuration to receiver receiver-1
    2014-01-14 15:29:54,412 [1389709794] Info :  [All] Dispatch OK of configuration to receiver receiver-1
    2014-01-14 15:29:54,413 [1389709794] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:54,415 [1389709794] Info :  I ask receiver-1 to wait a new conf
    2014-01-14 15:29:55,426 [1389709795] Info :  OK, all schedulers configurations are dispatched :)
    2014-01-14 15:29:55,426 [1389709795] Info :  [All] Trying to send configuration to receiver receiver-1
    2014-01-14 15:29:55,428 [1389709795] Info :  [All] Dispatch OK of configuration to receiver receiver-1
    2014-01-14 15:29:55,429 [1389709795] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:55,431 [1389709795] Info :  I ask receiver-1 to wait a new conf
    2014-01-14 15:29:56,442 [1389709796] Info :  OK, all schedulers configurations are dispatched :)
    2014-01-14 15:29:56,443 [1389709796] Info :  [All] Trying to send configuration to receiver receiver-1
    2014-01-14 15:29:56,445 [1389709796] Info :  [All] Dispatch OK of configuration to receiver receiver-1
    2014-01-14 15:29:56,446 [1389709796] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:56,447 [1389709796] Info :  I ask receiver-1 to wait a new conf
    2014-01-14 15:29:57,459 [1389709797] Info :  OK, all schedulers configurations are dispatched :)
    2014-01-14 15:29:57,459 [1389709797] Info :  [All] Trying to send configuration to receiver receiver-1
    2014-01-14 15:29:57,461 [1389709797] Info :  [All] Dispatch OK of configuration to receiver receiver-1
    2014-01-14 15:29:57,462 [1389709797] Info :  I ask poller-1 to wait a new conf
    2014-01-14 15:29:57,463 [1389709797] Info :  I ask receiver-1 to wait a new conf
    2014-01-14 15:29:58,479 [1389709798] Info :  OK, all schedulers configurations are dispatched :)
    2014-01-14 15:29:58,480 [1389709798] Info :  [All] Trying to send configuration to receiver receiver-1
    2014-01-14 15:29:58,481 [1389709798] Info :  [All] Dispatch OK of configuration to receiver receiver-1
    2014-01-14 15:29:58,483 [1389709798] Info :  I ask receiver-1 to wait a new conf
    2014-01-14 15:29:59,496 [1389709799] Info :  OK, all schedulers configurations are dispatched :)
    ...
    (and it continues like that)

    My questions are:
    (1) What is going there? Why does the arbiter tries to send a config to receiver-1 again and again, althoug there is every time an OK message too?
    (2) Also, the web interface on Server A is not reachable anymore - why that?

    Did I something wrong here or is there another reason for this behaviour?

    Regards,
    Juergen

  2. #2
    Junior Member
    Join Date
    May 2020
    Posts
    1
    Monitoring and all instrument is filled for the individuals. All the issues of the essay writers australia are involved for all shows. This reform is organized for the monitoring for the turns. Element is filed for the hope for all usefulness for the team.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •