Results 1 to 10 of 10

Thread: Shinken/Graphite on separate vmware esxi5 server [Errno 111] Connection refused

  1. 11-27-2012, 10:58 AM #1
    piellick
    piellick is offline
    Member
    Join Date
    Jun 2012
    Posts
    36

    Shinken/Graphite on separate vmware esxi5 server [Errno 111] Connection refused

    hi, firts post i'm honored 8).

    i've a strange problem with my configuration,

    2 vmware esxi5 servers with debian VM, check_esx3 works great (now ;D) on,1 esx server with shinken (debian squeeze vm) and 1 esx server with graphite (another debian squeeze vm).

    network looks good betweek graphite and shinken (dns,ping.. OK), i remove iptables from debian VM (no firewall)

    i configured Graphite broker on shinken-specific.cfg

    Code:
    define module {
 module_name Graphite-Perfdata
 module_type graphite_perfdata
 host  192.168.1.130 (my graphite VM ip)
 port 2003
}

define module {
 module_name GRAPHITE_UI
 module_type graphite_webui
 uri https://192.168.1.130 (my graphite VM ip)
 templates_path /usr/local/shinken/share/templates/graphite/
}

define broker {
 broker_name broker-1
 data_timeout 120
 timeout 3
 modules Livestatus,Simple-log,Graphite-Perfdata,WebUI
 manage_arbiters 1
 manage_sub_realms 1
 spare 0
 check_interval 60
 address localhost
 realm All
 max_check_attempts 3
 port 7772
}

    when i chek brokerd.log ...
    Code:
    2012-11-27 11:59:07,052 [1354013947] Error :  [broker-1] The instance Graphite-Perfdata raised an exception [Errno 111] Connection refused, I remove it!
2012-11-27 11:59:07,053 [1354013947] Error :  [broker-1] Back trace of this remove: Traceback (most recent call last):
 File "/usr/local/shinken/shinken/modulesmanager.py", line 131, in try_instance_init
  inst.init()
 File "/usr/local/shinken/shinken/modules/graphite_broker.py", line 82, in init
  self.con.connect((self.host, self.port))
 File "<string>", line 1, in connect
error: [Errno 111] Connection refused
    When i check on graphite UI, graph from my check (a check-esx3 plugin) are create ..but empty.

    Netstat from SHINKEN server :

    Code:
    root@shinken:/usr/local/shinken# netstat -tlpna
Connexions Internet actives (serveurs et établies)
Proto Recv-Q Send-Q Adresse locale     Adresse distante    Etat    PID/Program name
tcp    0   0 0.0.0.0:27017      0.0.0.0:*        LISTEN   1027/mongod
tcp    0   0 0.0.0.0:111       0.0.0.0:*        LISTEN   815/portmap
tcp    0   0 0.0.0.0:50000      0.0.0.0:*        LISTEN   3527/python
tcp    0   0 0.0.0.0:28017      0.0.0.0:*        LISTEN   1027/mongod
tcp    0   0 0.0.0.0:60562      0.0.0.0:*        LISTEN   827/rpc.statd
tcp    0   0 0.0.0.0:7766      0.0.0.0:*        LISTEN   3173/python
tcp    0   0 0.0.0.0:22       0.0.0.0:*        LISTEN   1718/sshd
tcp    0   0 0.0.0.0:7767      0.0.0.0:*        LISTEN   3536/python
tcp    0   0 127.0.0.1:631      0.0.0.0:*        LISTEN   1614/cupsd
tcp    0   0 0.0.0.0:7768      0.0.0.0:*        LISTEN   3229/python
tcp    0   0 0.0.0.0:7769      0.0.0.0:*        LISTEN   3315/python
tcp    0   0 127.0.0.1:25      0.0.0.0:*        LISTEN   1987/exim4
tcp    0   0 127.0.0.1:7770     0.0.0.0:*        LISTEN   3501/python
tcp    0   0 0.0.0.0:7771      0.0.0.0:*        LISTEN   3271/python
tcp    0   0 0.0.0.0:7772      0.0.0.0:*        LISTEN   3396/python
tcp    0   0 0.0.0.0:7773      0.0.0.0:*        LISTEN   3441/python
tcp    0   0 192.168.1.71:7767    192.168.1.10:60800   TIME_WAIT  -
tcp    0   0 127.0.0.1:56482     127.0.0.1:27017     ESTABLISHED 3536/python
tcp    0   0 127.0.0.1:7769     127.0.0.1:55444     ESTABLISHED 3315/python
tcp    0   0 127.0.0.1:7772     127.0.0.1:39698     ESTABLISHED 3396/python
tcp    0   0 127.0.0.1:27017     127.0.0.1:56449     ESTABLISHED 1027/mongod
tcp    0   0 127.0.0.1:44633     127.0.0.1:7771     ESTABLISHED 3396/python
tcp    0   0 127.0.0.1:56450     127.0.0.1:27017     ESTABLISHED 3208/python
tcp    0   0 192.168.1.71:37787   192.168.1.11:443    TIME_WAIT  -
tcp    0   0 127.0.0.1:39698     127.0.0.1:7772     ESTABLISHED 3501/python
tcp    0   0 127.0.0.1:45602     127.0.0.1:7768     ESTABLISHED 3271/python
tcp    0   0 127.0.0.1:7769     127.0.0.1:55434     ESTABLISHED 3315/python
tcp    0   0 127.0.0.1:7771     127.0.0.1:44626     ESTABLISHED 3271/python
tcp    0   0 127.0.0.1:27017     127.0.0.1:56450     ESTABLISHED 1027/mongod
tcp    0   0 127.0.0.1:27017     127.0.0.1:56482     ESTABLISHED 1027/mongod
tcp    0   0 127.0.0.1:45605     127.0.0.1:7768     ESTABLISHED 3396/python
tcp    0   0 192.168.1.71:7767    192.168.1.10:60801   TIME_WAIT  -
tcp    0   0 192.168.1.71:37790   192.168.1.11:443    TIME_WAIT  -
tcp    0   0 127.0.0.1:7768     127.0.0.1:45603     ESTABLISHED 3229/python
tcp    0   0 192.168.1.71:37788   192.168.1.11:443    TIME_WAIT  -
tcp    0   0 127.0.0.1:7768     127.0.0.1:45597     ESTABLISHED 3229/python
tcp    0   0 127.0.0.1:51795     127.0.0.1:7773     ESTABLISHED 3501/python
tcp    0   0 127.0.0.1:7768     127.0.0.1:45602     ESTABLISHED 3229/python
tcp    0   0 127.0.0.1:44626     127.0.0.1:7771     ESTABLISHED 3501/python
tcp    0   0 127.0.0.1:55444     127.0.0.1:7769     ESTABLISHED 3396/python
tcp    0   0 192.168.1.71:56764   192.168.1.130:2003   ESTABLISHED 3396/python
tcp    0   0 127.0.0.1:55434     127.0.0.1:7769     ESTABLISHED 3501/python
tcp    0   0 127.0.0.1:7773     127.0.0.1:51795     ESTABLISHED 3441/python
tcp    0   0 127.0.0.1:56449     127.0.0.1:27017     ESTABLISHED 3173/python
tcp    0   0 127.0.0.1:45603     127.0.0.1:7768     ESTABLISHED 3315/python
tcp    0   0 192.168.1.71:37791   192.168.1.11:443    TIME_WAIT  -
tcp    0   0 127.0.0.1:27017     127.0.0.1:56448     ESTABLISHED 1027/mongod
tcp    0   0 127.0.0.1:45597     127.0.0.1:7768     ESTABLISHED 3501/python
tcp    0   0 192.168.1.71:22     192.168.1.10:60693   ESTABLISHED 2876/1
tcp    0   0 192.168.1.71:37789   192.168.1.11:443    TIME_WAIT  -
tcp    0   0 127.0.0.1:56448     127.0.0.1:27017     ESTABLISHED 3204/python
tcp    0   0 127.0.0.1:7768     127.0.0.1:45605     ESTABLISHED 3229/python
tcp    0   0 127.0.0.1:7771     127.0.0.1:44633     ESTABLISHED 3271/python
tcp6    0   0 :::80          :::*          LISTEN   1498/apache2
tcp6    0   0 ::1:631         :::*          LISTEN   1614/cupsd
tcp6    0   0 ::1:25         :::*          LISTEN   1987/exim4
    Netstat from GRAPHITE server :

    Code:
    root@debian:~# netstat -tlpna
Connexions Internet actives (serveurs et établies)
Proto Recv-Q Send-Q Adresse locale     Adresse distante    Etat    PID/Program name
tcp    0   0 0.0.0.0:111       0.0.0.0:*        LISTEN   828/portmap
tcp    0   0 0.0.0.0:2003      0.0.0.0:*        LISTEN   1829/python
tcp    0   0 0.0.0.0:2004      0.0.0.0:*        LISTEN   1829/python
tcp    0   0 0.0.0.0:60661      0.0.0.0:*        LISTEN   840/rpc.statd
tcp    0   0 0.0.0.0:22       0.0.0.0:*        LISTEN   1172/sshd
tcp    0   0 127.0.0.1:631      0.0.0.0:*        LISTEN   1179/cupsd
tcp    0   0 127.0.0.1:25      0.0.0.0:*        LISTEN   1615/exim4
tcp    0   0 0.0.0.0:7002      0.0.0.0:*        LISTEN   1829/python
tcp    0   0 127.0.0.1:34826     127.0.0.1:7002     ESTABLISHED 1141/apache2
tcp    0   0 192.168.1.130:2003   192.168.1.71:56764   ESTABLISHED 1829/python
tcp    0   0 127.0.0.1:7002     127.0.0.1:34826     ESTABLISHED 1829/python
tcp    0   52 192.168.1.130:22    192.168.1.10:60776   ESTABLISHED 1875/1
tcp6    0   0 :::80          :::*          LISTEN   1065/apache2
tcp6    0   0 :::22          :::*          LISTEN   1172/sshd
tcp6    0   0 ::1:631         :::*          LISTEN   1179/cupsd
tcp6    0   0 ::1:25         :::*          LISTEN   1615/exim4
tcp6    0   0 :::443         :::*          LISTEN   1065/apache2
    graphite looks listening on good port ...

    logs from graphite creater
    Code:
    oot@debian:~# vi /opt/graphite/storage/log/carbon-cache/carbon-cache-a/creates.log
18/12/2012 16:44:10 :: new metric esx1.Mem.mem_usage matched schema default_1min_for_1day
18/12/2012 16:44:10 :: new metric esx1.Mem.mem_usage matched aggregation schema default
18/12/2012 16:44:10 :: creating database file /opt/graphite/storage/whisper/esx1/Mem/mem_usage.wsp (archive=[(60, 1440)] xff=None agg=None)
18/12/2012 16:44:10 :: new metric esx1.Mem.mem_memctl matched schema default_1min_for_1day
18/12/2012 16:44:10 :: new metric esx1.Mem.mem_memctl matched aggregation schema default
18/12/2012 16:44:10 :: creating database file /opt/graphite/storage/whisper/esx1/Mem/mem_memctl.wsp (archive=[(60, 1440)] xff=None agg=None)
18/12/2012 16:44:10 :: new metric esx1.Mem.mem_swap matched schema default_1min_for_1day
18/12/2012 16:44:10 :: new metric esx1.Mem.mem_swap matched aggregation schema default
18/12/2012 16:44:10 :: creating database file /opt/graphite/storage/whisper/esx1/Mem/mem_swap.wsp (archive=[(60, 1440)] xff=None agg=None)
18/12/2012 16:44:10 :: new metric esx1.Cpu.cpu_usagemhz matched schema default_1min_for_1day
18/12/2012 16:44:10 :: new metric esx1.Cpu.cpu_usagemhz matched aggregation schema default
18/12/2012 16:44:10 :: creating database file /opt/graphite/storage/whisper/esx1/Cpu/cpu_usagemhz.wsp (archive=[(60, 1440)] xff=None agg=None)
18/12/2012 16:44:11 :: new metric esx1.Net.net_receive matched schema default_1min_for_1day
18/12/2012 16:44:11 :: new metric esx1.Net.net_receive matched aggregation schema default
18/12/2012 16:44:11 :: creating database file /opt/graphite/storage/whisper/esx1/Net/net_receive.wsp (archive=[(60, 1440)] xff=None agg=None)
18/12/2012 16:44:11 :: new metric esx1.Net.Bad_NICs matched schema default_1min_for_1day
18/12/2012 16:44:11 :: new metric esx1.Net.Bad_NICs matched aggregation schema default
18/12/2012 16:44:11 :: creating database file /opt/graphite/storage/whisper/esx1/Net/Bad_NICs.wsp (archive=[(60, 1440)] xff=None agg=None)
18/12/2012 16:44:11 :: new metric esx1.Net.net_send matched schema default_1min_for_1day
18/12/2012 16:44:11 :: new metric esx1.Net.net_send matched aggregation schema default
18/12/2012 16:44:11 :: creating database file /opt/graphite/storage/whisper/esx1/Net/net_send.wsp (archive=[(60, 1440)] xff=None agg=None)
18/12/2012 16:44:11 :: new metric esx1.Io.io_aborted matched schema default_1min_for_1day
18/12/2012 16:44:11 :: new metric esx1.Io.io_aborted matched aggregation schema default
18/12/2012 16:44:11 :: creating database file /opt/graphite/storage/whisper/esx1/Io/io_aborted.wsp (archive=[(60, 1440)] xff=None agg=None)
18/12/2012 16:44:11 :: new metric esx1.Io.io_write matched schema default_1min_for_1day
18/12/2012 16:44:11 :: new metric esx1.Io.io_write matched aggregation schema default
18/12/2012 16:44:11 :: creating database file /opt/graphite/storage/whisper/esx1/Io/io_write.wsp (archive=[(60, 1440)] xff=None agg=None)
18/12/2012 16:44:11 :: new metric esx1.Io.io_busresets matched schema default_1min_for_1day
18/12/2012 16:44:11 :: new metric esx1.Io.io_busresets matched aggregation schema default
18/12/2012 16:44:11 :: creating database file /opt/graphite/storage/whisper/esx1/Io/io_busresets.wsp (archive=[(60, 1440)] xff=None agg=None)
18/12/2012 16:44:11 :: new metric esx1.Io.io_device matched schema default_1min_for_1day
18/12/2012 16:44:11 :: new metric esx1.Io.io_device matched aggregation schema default
18/12/2012 16:44:11 :: creating database file /opt/graphite/storage/whisper/esx1/Io/io_device.wsp (archive=[(60, 1440)] xff=None agg=None)
18/12/2012 16:44:11 :: new metric esx1.Net.OK_NICs matched schema default_1min_for_1day
18/12/2012 16:44:11 :: new metric esx1.Net.OK_NICs matched aggregation schema default
18/12/2012 16:44:11 :: creating database file /opt/graphite/storage/whisper/esx1/Net/OK_NICs.wsp (archive=[(60, 1440)] xff=None agg=None)
18/12/2012 16:44:11 :: new metric esx1.Io.io_kernel matched schema default_1min_for_1day
18/12/2012 16:44:11 :: new metric esx1.Io.io_kernel matched aggregation schema default
18/12/2012 16:44:11 :: creating database file /opt/graphite/storage/whisper/esx1/Io/io_kernel.wsp (archive=[(60, 1440)] xff=None agg=None)
18/12/2012 16:44:11 :: new metric esx1.Io.io_read matched schema default_1min_for_1day
18/12/2012 16:44:11 :: new metric esx1.Io.io_read matched aggregation schema default
18/12/2012 16:44:12 :: creating database file /opt/graphite/storage/whisper/esx1/Io/io_read.wsp (archive=[(60, 1440)] xff=None agg=None)
18/12/2012 16:44:12 :: new metric esx1.Io.io_queue matched schema default_1min_for_1day
18/12/2012 16:44:12 :: new metric esx1.Io.io_queue matched aggregation schema default
18/12/2012 16:44:12 :: creating database file /opt/graphite/storage/whisper/esx1/Io/io_queue.wsp (archive=[(60, 1440)] xff=None agg=None)
    logs from graphite listener

    Code:
    ....
18/12/2012 15:30:28 :: MetricLineReceiver connection with 192.168.1.71:40454 established
18/12/2012 15:32:58 :: MetricLineReceiver connection with 192.168.1.71:40454 closed cleanly
18/12/2012 15:33:06 :: MetricLineReceiver connection with 192.168.1.71:40497 established
18/12/2012 15:39:34 :: MetricLineReceiver connection with 192.168.1.71:40497 closed cleanly
18/12/2012 16:07:41 :: MetricLineReceiver connection with 192.168.1.71:56764 established
18/12/2012 16:16:07 :: MetricLineReceiver connection with 192.168.1.71:56764 closed cleanly
18/12/2012 16:16:15 :: MetricLineReceiver connection with 192.168.1.71:56828 established
18/12/2012 16:19:09 :: MetricLineReceiver connection with 192.168.1.71:56828 closed cleanly
18/12/2012 16:19:17 :: MetricLineReceiver connection with 192.168.1.71:56886 established
18/12/2012 16:28:23 :: MetricLineReceiver connection with 192.168.1.71:56886 closed cleanly
18/12/2012 16:28:24 :: MetricLineReceiver connection with 192.168.1.71:56953 established
18/12/2012 16:31:10 :: MetricLineReceiver connection with 192.168.1.71:56953 closed cleanly
18/12/2012 16:31:19 :: MetricLineReceiver connection with 192.168.1.71:57026 established
18/12/2012 16:37:46 :: MetricLineReceiver connection with 192.168.1.71:57026 closed cleanly
18/12/2012 16:37:54 :: MetricLineReceiver connection with 192.168.1.71:57094 established
18/12/2012 16:43:35 :: MetricLineReceiver connection with 192.168.1.71:57094 closed cleanly
18/12/2012 16:43:43 :: MetricLineReceiver connection with 192.168.1.71:57168 established

    Someone have an idea ?



    Reply With Quote Reply With Quote
  2. 11-27-2012, 12:43 PM #2
    Seb-Solon
    Seb-Solon is offline
    Administrator
    Join Date
    Jun 2011
    Posts
    216

    Re: Shinken/Graphite on separate vmware esxi5 server [Errno 111] Connection refused

    Well it's strange. The netstat returns that tcp connexion is established. Maybe from a previous instance of Shinken?

    Can you try to stop Shinekn and see if the port switch to LISTEN?
    Reply With Quote Reply With Quote
  3. 11-27-2012, 12:58 PM #3
    piellick
    piellick is offline
    Member
    Join Date
    Jun 2012
    Posts
    36

    Re: Shinken/Graphite on separate vmware esxi5 server [Errno 111] Connection refused

    very strange yes... i stopped shinken (service shinken stop) and the connection is now 'TIME_WAIT" :

    Code:
    root@shinken:/usr/local/shinken# netstat -tlpna
Connexions Internet actives (serveurs et établies)
Proto Recv-Q Send-Q Adresse locale     Adresse distante    Etat    PID/Program name
tcp    0   0 0.0.0.0:27017      0.0.0.0:*        LISTEN   1027/mongod
tcp    0   0 0.0.0.0:111       0.0.0.0:*        LISTEN   815/portmap
tcp    0   0 0.0.0.0:28017      0.0.0.0:*        LISTEN   1027/mongod
tcp    0   0 0.0.0.0:60562      0.0.0.0:*        LISTEN   827/rpc.statd
tcp    0   0 0.0.0.0:22       0.0.0.0:*        LISTEN   1718/sshd
tcp    0   0 127.0.0.1:631      0.0.0.0:*        LISTEN   1614/cupsd
tcp    0   0 127.0.0.1:25      0.0.0.0:*        LISTEN   1987/exim4
tcp    0   0 127.0.0.1:57418     127.0.0.1:27017     TIME_WAIT  -
tcp    0   0 192.168.1.71:57700   192.168.1.130:2003   TIME_WAIT  -
tcp    0   52 192.168.1.71:22     192.168.1.10:60693   ESTABLISHED 2876/1
tcp6    0   0 :::80          :::*          LISTEN   1498/apache2
tcp6    0   0 ::1:631         :::*          LISTEN   1614/cupsd
tcp6    0   0 ::1:25         :::*          LISTEN   1987/exim4
tcp6    0   0 192.168.1.71:80     192.168.1.10:63051   TIME_WAIT  -
tcp6    0   0 192.168.1.71:80     192.168.1.10:63052   ESTABLISHED 7912/apache2
root@shinken:/usr/local/shinken#

    I restart shinken and connection will be "ETABLISHED":
    Code:
    tcp    0   0 192.168.1.71:57839   192.168.1.130:2003   ESTABLISHED 10534/python
    it's should work ...but not :

    An other information :

    Graph from localhost graph agent (carbon.agents.MYHOST.*) works great... i think that graphite configuration is right..
    Reply With Quote Reply With Quote
  4. 11-28-2012, 05:49 AM #4
    piellick
    piellick is offline
    Member
    Join Date
    Jun 2012
    Posts
    36

    Re: Shinken/Graphite on separate vmware esxi5 server [Errno 111] Connection refused

    There is a solution for debug "Graphite-Perfdata " module ? then i post it on the forum.

    thanks
    Reply With Quote Reply With Quote
  5. 11-28-2012, 06:10 AM #5
    xkilian
    xkilian is offline
    Administrator
    Join Date
    Dec 2011
    Posts
    278

    Re: Shinken/Graphite on separate vmware esxi5 server [Errno 111] Connection refused

    Ahhh. Interesting problems. :-)

    I suggest you:

    [list type=decimal]
    [li]Do a network trace on the Shinken server using tcpdump: tcpdump host 192.168.1.130 -w packet_trace.pcap[/li]
    [li]Open the trace file with Wireshark, look at the communications. You should see the connection failure or timeout probably accompanie with an ICMP control packet with some indication of what went wrong.[/li]
    [li]You can also add debug statements, but I do not think they will do you any good, perhaps at least to list the startup options.[/li][/list]

    If you do not know wat to do with it, put it up and I will take a look. Make sure you capture the traffic when the connection timeout occurs.

    Your configuration looks good. You can add debug statements, but the fail happens when it creates a connection. It will try and reconnect later. Which is why it still works.

    Cheers,

    xkilian
    Reply With Quote Reply With Quote
  6. 11-28-2012, 06:53 AM #6
    piellick
    piellick is offline
    Member
    Join Date
    Jun 2012
    Posts
    36

    Re: Shinken/Graphite on separate vmware esxi5 server [Errno 111] Connection refused

    hi, good idea !

    there is a dump from my shinken server :

    Code:
    07:51:08.833476 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.71 tell 192.168.1.130, length 46
07:51:08.833496 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.1.71 is-at 00:0c:29:8f:52:05 (oui Unknown), length 28
07:51:38.310073 IP (tos 0x0, ttl 64, id 46469, offset 0, flags [DF], proto TCP (6), length 214)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0x84e2 (incorrect -> 0x51a7), seq 9476:9638, ack 1, win 92, options [nop,nop,TS val 17937628 ecr 17938878], length 162
07:51:38.311043 IP (tos 0x0, ttl 64, id 8189, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x82cc (correct), seq 1, ack 9638, win 905, options [nop,nop,TS val 17946479 ecr 17937628], length 0
07:51:38.312345 IP (tos 0x0, ttl 64, id 46470, offset 0, flags [DF], proto TCP (6), length 238)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0x84fa (incorrect -> 0x371d), seq 9638:9824, ack 1, win 92, options [nop,nop,TS val 17937629 ecr 17946479], length 186
07:51:38.313375 IP (tos 0x0, ttl 64, id 8190, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x81e2 (correct), seq 1, ack 9824, win 951, options [nop,nop,TS val 17946480 ecr 17937629], length 0
07:51:38.313397 IP (tos 0x0, ttl 64, id 46471, offset 0, flags [DF], proto TCP (6), length 259)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0x850f (incorrect -> 0xd385), seq 9824:10031, ack 1, win 92, options [nop,nop,TS val 17937629 ecr 17946480], length 207
07:51:38.314742 IP (tos 0x0, ttl 64, id 8191, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x80e6 (correct), seq 1, ack 10031, win 996, options [nop,nop,TS val 17946480 ecr 17937629], length 0
07:51:39.327740 IP (tos 0x0, ttl 64, id 46472, offset 0, flags [DF], proto TCP (6), length 159)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0x84ab (incorrect -> 0x54bd), seq 10031:10138, ack 1, win 92, options [nop,nop,TS val 17937882 ecr 17946480], length 107
07:51:39.328418 IP (tos 0x0, ttl 64, id 8192, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x7e81 (correct), seq 1, ack 10138, win 996, options [nop,nop,TS val 17946733 ecr 17937882], length 0
07:51:39.328485 IP (tos 0x0, ttl 64, id 46473, offset 0, flags [DF], proto TCP (6), length 372)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0x8580 (incorrect -> 0xf8f8), seq 10138:10458, ack 1, win 92, options [nop,nop,TS val 17937883 ecr 17946733], length 320
07:51:39.329013 IP (tos 0x0, ttl 64, id 8193, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x7d3a (correct), seq 1, ack 10458, win 1002, options [nop,nop,TS val 17946733 ecr 17937883], length 0
07:51:39.329032 IP (tos 0x0, ttl 64, id 46474, offset 0, flags [DF], proto TCP (6), length 129)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0x848d (incorrect -> 0x70d1), seq 10458:10535, ack 1, win 92, options [nop,nop,TS val 17937883 ecr 17946733], length 77
07:51:39.331864 IP (tos 0x0, ttl 64, id 8194, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x7cec (correct), seq 1, ack 10535, win 1002, options [nop,nop,TS val 17946734 ecr 17937883], length 0
07:51:40.338560 IP (tos 0x0, ttl 64, id 46475, offset 0, flags [DF], proto TCP (6), length 162)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0x84ae (incorrect -> 0xb0d3), seq 10535:10645, ack 1, win 92, options [nop,nop,TS val 17938135 ecr 17946734], length 110
07:51:40.339918 IP (tos 0x0, ttl 64, id 8195, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x7a86 (correct), seq 1, ack 10645, win 1002, options [nop,nop,TS val 17946986 ecr 17938135], length 0
07:51:40.339944 IP (tos 0x0, ttl 64, id 46476, offset 0, flags [DF], proto TCP (6), length 432)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0x85bc (incorrect -> 0x7abf), seq 10645:11025, ack 1, win 92, options [nop,nop,TS val 17938135 ecr 17946986], length 380
07:51:40.340976 IP (tos 0x0, ttl 64, id 8196, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x790a (correct), seq 1, ack 11025, win 1002, options [nop,nop,TS val 17946986 ecr 17938135], length 0
07:51:40.341007 IP (tos 0x0, ttl 64, id 46477, offset 0, flags [DF], proto TCP (6), length 457)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0x85d5 (incorrect -> 0xe4d4), seq 11025:11430, ack 1, win 92, options [nop,nop,TS val 17938136 ecr 17946986], length 405
07:51:40.342356 IP (tos 0x0, ttl 64, id 8197, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x7773 (correct), seq 1, ack 11430, win 1002, options [nop,nop,TS val 17946987 ecr 17938136], length 0
07:51:41.354309 IP (tos 0x0, ttl 64, id 46478, offset 0, flags [DF], proto TCP (6), length 378)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0x8586 (incorrect -> 0x65b3), seq 11430:11756, ack 1, win 92, options [nop,nop,TS val 17938389 ecr 17946987], length 326
07:51:41.354946 IP (tos 0x0, ttl 64, id 8198, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x7433 (correct), seq 1, ack 11756, win 1002, options [nop,nop,TS val 17947240 ecr 17938389], length 0
07:51:41.354967 IP (tos 0x0, ttl 64, id 46479, offset 0, flags [DF], proto TCP (6), length 131)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0x848f (incorrect -> 0xbb6b), seq 11756:11835, ack 1, win 92, options [nop,nop,TS val 17938389 ecr 17947240], length 79
07:51:41.356031 IP (tos 0x0, ttl 64, id 8199, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x73e4 (correct), seq 1, ack 11835, win 1002, options [nop,nop,TS val 17947240 ecr 17938389], length 0
07:51:41.356055 IP (tos 0x0, ttl 64, id 46480, offset 0, flags [DF], proto TCP (6), length 261)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0x8511 (incorrect -> 0x748a), seq 11835:12044, ack 1, win 92, options [nop,nop,TS val 17938389 ecr 17947240], length 209
07:51:41.357241 IP (tos 0x0, ttl 64, id 8200, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x7312 (correct), seq 1, ack 12044, win 1002, options [nop,nop,TS val 17947241 ecr 17938389], length 0
07:51:43.309261 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.71 tell 192.168.1.130, length 46
07:51:43.309288 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.1.71 is-at 00:0c:29:8f:52:05 (oui Unknown), length 28
07:54:45.006005 IP (tos 0x0, ttl 64, id 46481, offset 0, flags [DF], proto TCP (6), length 1500)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [.], cksum 0x89e8 (incorrect -> 0xee56), seq 12044:13492, ack 1, win 92, options [nop,nop,TS val 17984302 ecr 17947241], length 1448
07:54:45.006025 IP (tos 0x0, ttl 64, id 46482, offset 0, flags [DF], proto TCP (6), length 220)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0x84e8 (incorrect -> 0x03f9), seq 13492:13660, ack 1, win 92, options [nop,nop,TS val 17984302 ecr 17947241], length 168
07:54:45.008698 IP (tos 0x0, ttl 64, id 8201, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x06b7 (correct), seq 1, ack 13492, win 1002, options [nop,nop,TS val 17993154 ecr 17984302], length 0
07:54:45.008776 IP (tos 0x0, ttl 64, id 8202, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x0611 (correct), seq 1, ack 13660, win 1000, options [nop,nop,TS val 17993154 ecr 17984302], length 0
07:54:45.008792 IP (tos 0x0, ttl 64, id 46483, offset 0, flags [DF], proto TCP (6), length 150)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0x84a2 (incorrect -> 0xf2d6), seq 13660:13758, ack 1, win 92, options [nop,nop,TS val 17984302 ecr 17993154], length 98
07:54:45.010307 IP (tos 0x0, ttl 64, id 8203, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x05ad (correct), seq 1, ack 13758, win 1002, options [nop,nop,TS val 17993154 ecr 17984302], length 0
07:54:48.043003 IP (tos 0x0, ttl 64, id 46484, offset 0, flags [DF], proto TCP (6), length 1500)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [.], cksum 0x89e8 (incorrect -> 0xa827), seq 13758:15206, ack 1, win 92, options [nop,nop,TS val 17985061 ecr 17993154], length 1448
07:54:48.043022 IP (tos 0x0, ttl 64, id 46485, offset 0, flags [DF], proto TCP (6), length 1082)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0x8846 (incorrect -> 0xf22b), seq 15206:16236, ack 1, win 92, options [nop,nop,TS val 17985061 ecr 17993154], length 1030
07:54:48.044671 IP (tos 0x0, ttl 64, id 8204, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0xfa16 (correct), seq 1, ack 15206, win 1002, options [nop,nop,TS val 17993913 ecr 17985061], length 0
07:54:48.044704 IP (tos 0x0, ttl 64, id 8205, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0xf620 (correct), seq 1, ack 16236, win 986, options [nop,nop,TS val 17993913 ecr 17985061], length 0
07:54:48.044720 IP (tos 0x0, ttl 64, id 46486, offset 0, flags [DF], proto TCP (6), length 187)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0x84c7 (incorrect -> 0x4e68), seq 16236:16371, ack 1, win 92, options [nop,nop,TS val 17985061 ecr 17993913], length 135
07:54:48.046631 IP (tos 0x0, ttl 64, id 8206, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0xf589 (correct), seq 1, ack 16371, win 1002, options [nop,nop,TS val 17993913 ecr 17985061], length 0
07:54:50.008302 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.71 tell 192.168.1.130, length 46
07:54:50.008321 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.1.71 is-at 00:0c:29:8f:52:05 (oui Unknown), length 28
    i have bad checksum for 1/2 IP packet, very strange.... I'll investigate side of my networks interface from Esx Server. : Ore something else ?

    pierrick
    Reply With Quote Reply With Quote
  7. 11-28-2012, 07:06 AM #7
    piellick
    piellick is offline
    Member
    Join Date
    Jun 2012
    Posts
    36

    Re: Shinken/Graphite on separate vmware esxi5 server [Errno 111] Connection refused

    Ignore Bad Cheksum ... (or not ?)
    http://kb.vmware.com/selfservice/mic...rnalId=1003325

    the network traffic is captured before the checksum is calculated and, therefore, the checksum is incorrect.


    The same dump this time on graphite side :

    Code:
    12:13:43.594248 IP (tos 0x0, ttl 64, id 46607, offset 0, flags [DF], proto TCP (6), length 215)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0x5ab1 (correct), seq 4329:4492, ack 1, win 92, options [nop,nop,TS val 18338002 ecr 18339749], length 163
12:13:43.594306 IP (tos 0x0, ttl 64, id 8327, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x8de8 (correct), seq 1, ack 4492, win 1002, options [nop,nop,TS val 18346855 ecr 18338002], length 0
12:13:43.594899 IP (tos 0x0, ttl 64, id 46608, offset 0, flags [DF], proto TCP (6), length 159)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0x5fc2 (correct), seq 4492:4599, ack 1, win 92, options [nop,nop,TS val 18338002 ecr 18346855], length 107
12:13:43.595002 IP (tos 0x0, ttl 64, id 8328, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x8d7d (correct), seq 1, ack 4599, win 1002, options [nop,nop,TS val 18346855 ecr 18338002], length 0
12:13:43.597161 IP (tos 0x0, ttl 64, id 46609, offset 0, flags [DF], proto TCP (6), length 129)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0x7f11 (correct), seq 4599:4676, ack 1, win 92, options [nop,nop,TS val 18338003 ecr 18346855], length 77
12:13:43.597176 IP (tos 0x0, ttl 64, id 8329, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x8d2e (correct), seq 1, ack 4676, win 1002, options [nop,nop,TS val 18346856 ecr 18338003], length 0
12:13:43.598580 IP (tos 0x0, ttl 64, id 46610, offset 0, flags [DF], proto TCP (6), length 238)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0x4dde (correct), seq 4676:4862, ack 1, win 92, options [nop,nop,TS val 18338003 ecr 18346856], length 186
12:13:43.598592 IP (tos 0x0, ttl 64, id 8330, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x8c74 (correct), seq 1, ack 4862, win 1002, options [nop,nop,TS val 18346856 ecr 18338003], length 0
12:13:45.662280 IP (tos 0x0, ttl 64, id 46611, offset 0, flags [DF], proto TCP (6), length 238)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0x411f (correct), seq 4862:5048, ack 1, win 92, options [nop,nop,TS val 18338519 ecr 18346856], length 186
12:13:45.662303 IP (tos 0x0, ttl 64, id 8331, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x87b2 (correct), seq 1, ack 5048, win 1002, options [nop,nop,TS val 18347372 ecr 18338519], length 0
12:13:45.663207 IP (tos 0x0, ttl 64, id 46612, offset 0, flags [DF], proto TCP (6), length 428)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0xcca6 (correct), seq 5048:5424, ack 1, win 92, options [nop,nop,TS val 18338519 ecr 18347372], length 376
12:13:45.663226 IP (tos 0x0, ttl 64, id 8332, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x8639 (correct), seq 1, ack 5424, win 1002, options [nop,nop,TS val 18347373 ecr 18338519], length 0
12:13:46.674674 IP (tos 0x0, ttl 64, id 46613, offset 0, flags [DF], proto TCP (6), length 184)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0xa4fe (correct), seq 5424:5556, ack 1, win 92, options [nop,nop,TS val 18338772 ecr 18347373], length 132
12:13:46.674702 IP (tos 0x0, ttl 64, id 8333, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x83bc (correct), seq 1, ack 5556, win 1002, options [nop,nop,TS val 18347625 ecr 18338772], length 0
12:13:46.675642 IP (tos 0x0, ttl 64, id 46614, offset 0, flags [DF], proto TCP (6), length 379)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0xa29c (correct), seq 5556:5883, ack 1, win 92, options [nop,nop,TS val 18338773 ecr 18347625], length 327
12:13:46.675656 IP (tos 0x0, ttl 64, id 8334, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x8273 (correct), seq 1, ack 5883, win 1002, options [nop,nop,TS val 18347626 ecr 18338773], length 0
12:13:46.677196 IP (tos 0x0, ttl 64, id 46615, offset 0, flags [DF], proto TCP (6), length 262)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0x164d (correct), seq 5883:6093, ack 1, win 92, options [nop,nop,TS val 18338773 ecr 18347626], length 210
12:13:46.677210 IP (tos 0x0, ttl 64, id 8335, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x81a1 (correct), seq 1, ack 6093, win 1002, options [nop,nop,TS val 18347626 ecr 18338773], length 0
12:13:46.678210 IP (tos 0x0, ttl 64, id 46616, offset 0, flags [DF], proto TCP (6), length 346)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0xd0e2 (correct), seq 6093:6387, ack 1, win 92, options [nop,nop,TS val 18338773 ecr 18347626], length 294
12:13:46.678223 IP (tos 0x0, ttl 64, id 8336, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x807b (correct), seq 1, ack 6387, win 1002, options [nop,nop,TS val 18347626 ecr 18338773], length 0
12:13:47.684720 IP (tos 0x0, ttl 64, id 46617, offset 0, flags [DF], proto TCP (6), length 162)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0xb162 (correct), seq 6387:6497, ack 1, win 92, options [nop,nop,TS val 18339025 ecr 18347626], length 110
12:13:47.684744 IP (tos 0x0, ttl 64, id 8337, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x7e15 (correct), seq 1, ack 6497, win 1002, options [nop,nop,TS val 18347878 ecr 18339025], length 0
12:13:47.686158 IP (tos 0x0, ttl 64, id 46618, offset 0, flags [DF], proto TCP (6), length 131)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0xc14f (correct), seq 6497:6576, ack 1, win 92, options [nop,nop,TS val 18339025 ecr 18347878], length 79
12:13:47.686171 IP (tos 0x0, ttl 64, id 8338, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x7dc6 (correct), seq 1, ack 6576, win 1002, options [nop,nop,TS val 18347878 ecr 18339025], length 0
12:13:47.686593 IP (tos 0x0, ttl 64, id 46619, offset 0, flags [DF], proto TCP (6), length 372)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0xe92a (correct), seq 6576:6896, ack 1, win 92, options [nop,nop,TS val 18339025 ecr 18347878], length 320
12:13:47.686605 IP (tos 0x0, ttl 64, id 8339, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x7c86 (correct), seq 1, ack 6896, win 1002, options [nop,nop,TS val 18347878 ecr 18339025], length 0
12:13:51.739734 IP (tos 0x0, ttl 64, id 46620, offset 0, flags [DF], proto TCP (6), length 187)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0xdff3 (correct), seq 6896:7031, ack 1, win 92, options [nop,nop,TS val 18340039 ecr 18347878], length 135
12:13:51.739757 IP (tos 0x0, ttl 64, id 8340, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x7413 (correct), seq 1, ack 7031, win 1002, options [nop,nop,TS val 18348892 ecr 18340039], length 0
12:13:51.742402 IP (tos 0x0, ttl 64, id 46621, offset 0, flags [DF], proto TCP (6), length 1500)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [.], cksum 0x7fdc (correct), seq 7031:8479, ack 1, win 92, options [nop,nop,TS val 18340039 ecr 18348892], length 1448
12:13:51.742416 IP (tos 0x0, ttl 64, id 8341, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x6e6b (correct), seq 1, ack 8479, win 1002, options [nop,nop,TS val 18348892 ecr 18340039], length 0
12:13:51.742464 IP (tos 0x0, ttl 64, id 46622, offset 0, flags [DF], proto TCP (6), length 1082)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0xa5ce (correct), seq 8479:9509, ack 1, win 92, options [nop,nop,TS val 18340039 ecr 18348892], length 1030
12:13:51.742473 IP (tos 0x0, ttl 64, id 8342, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x6a75 (correct), seq 1, ack 9509, win 986, options [nop,nop,TS val 18348892 ecr 18340039], length 0
12:13:53.766288 IP (tos 0x0, ttl 64, id 46623, offset 0, flags [DF], proto TCP (6), length 150)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0x573f (correct), seq 9509:9607, ack 1, win 92, options [nop,nop,TS val 18340545 ecr 18348892], length 98
12:13:53.766341 IP (tos 0x0, ttl 64, id 8343, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x660f (correct), seq 1, ack 9607, win 1002, options [nop,nop,TS val 18349398 ecr 18340545], length 0
12:13:54.781490 IP (tos 0x0, ttl 64, id 46624, offset 0, flags [DF], proto TCP (6), length 1500)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [.], cksum 0x860e (correct), seq 9607:11055, ack 1, win 92, options [nop,nop,TS val 18340799 ecr 18349398], length 1448
12:13:54.781513 IP (tos 0x0, ttl 64, id 8344, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x5e6b (correct), seq 1, ack 11055, win 1002, options [nop,nop,TS val 18349652 ecr 18340799], length 0
12:13:54.781610 IP (tos 0x0, ttl 64, id 46625, offset 0, flags [DF], proto TCP (6), length 218)
  192.168.1.71.39297 > 192.168.1.130.cfinger: Flags [P.], cksum 0x823d (correct), seq 11055:11221, ack 1, win 92, options [nop,nop,TS val 18340799 ecr 18349398], length 166
12:13:54.781619 IP (tos 0x0, ttl 64, id 8345, offset 0, flags [DF], proto TCP (6), length 52)
  192.168.1.130.cfinger > 192.168.1.71.39297: Flags [.], cksum 0x5dc7 (correct), seq 1, ack 11221, win 1000, options [nop,nop,TS val 18349652 ecr 18340799], length 0
12:13:56.738976 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.71 tell 192.168.1.130, length 28
12:13:56.739684 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.1.71 is-at 00:0c:29:8f:52:05 (oui Unknown), length 46
    no cheksum error ... but same process using ... (tcpdump) ???

    pierrick
    Reply With Quote Reply With Quote
  8. 11-28-2012, 11:19 AM #8
    piellick
    piellick is offline
    Member
    Join Date
    Jun 2012
    Posts
    36

    Re: Shinken/Graphite on separate vmware esxi5 server [Errno 111] Connection refused

    more investigation (I want to clarify that my check looks always good and WebUI works great .. :-\).

    logs from arbiterd.log
    Code:
    2012-11-28 11:35:42,928 [1354098942] Warning : Printing stored debug messages prior to our daemonization
2012-11-28 11:35:52,593 [1354098952] Error :  Failed sending configuration for receiver-1: receiving: connection lost: [Errno 104] Connection reset by peer
2012-11-28 11:35:52,600 [1354098952] Error :  [All] Dispatching failed for receiver receiver-1
2012-11-28 11:35:58,448 [1354098958] Warning : Printing stored debug messages prior to our daemonization
2012-11-28 11:44:27,085 [1354099467] Critical : I got an unrecoverable error. I have to exit
2012-11-28 11:44:27,088 [1354099467] Critical : You can log a bug ticket at https://github.com/naparuba/shinken/issues/new to get help
2012-11-28 11:44:27,100 [1354099467] Critical : Back trace of it: Traceback (most recent call last):
 File "/usr/local/shinken/shinken/daemons/skonfdaemon.py", line 442, in main
  self.do_mainloop()
 File "/usr/local/shinken/shinken/daemon.py", line 244, in do_mainloop
  self.do_loop_turn()
 File "/usr/local/shinken/shinken/daemons/skonfdaemon.py", line 463, in do_loop_turn
  self.run()
 File "/usr/local/shinken/shinken/daemons/skonfdaemon.py", line 566, in run
  srv = run(host=self.http_host, port=self.http_port, server=self.http_backend)
 File "/usr/local/shinken/shinken/webui/bottle.py", line 2203, in run
  res = server.run(app)
 File "/usr/local/shinken/shinken/webui/bottle.py", line 2088, in run
  return sa(self.host, self.port, **self.options).run(handler)
 File "/usr/local/shinken/shinken/webui/bottle.py", line 1907, in run
  srv.serve_forever()
 File "/usr/lib/python2.6/SocketServer.py", line 224, in serve_forever
  r, w, e = select.select([self], [], [], poll_interval)
error: (4, 'Interrupted system call')
2012-11-28 11:46:50,585 [1354099610] Warning : Printing stored debug messages prior to our daemonization
2012-11-28 11:46:54,736 [1354099614] Warning : Printing stored debug messages prior to our daemonization
2012-11-28 11:47:10,525 [1354099630] Critical : I got an unrecoverable error. I have to exit
2012-11-28 11:47:10,526 [1354099630] Critical : You can log a bug ticket at https://github.com/naparuba/shinken/issues/new to get help
2012-11-28 11:47:10,541 [1354099630] Critical : Back trace of it: Traceback (most recent call last):
 File "/usr/local/shinken/shinken/daemons/skonfdaemon.py", line 442, in main
  self.do_mainloop()
 File "/usr/local/shinken/shinken/daemon.py", line 244, in do_mainloop
  self.do_loop_turn()
 File "/usr/local/shinken/shinken/daemons/skonfdaemon.py", line 463, in do_loop_turn
  self.run()
 File "/usr/local/shinken/shinken/daemons/skonfdaemon.py", line 566, in run
  srv = run(host=self.http_host, port=self.http_port, server=self.http_backend)
 File "/usr/local/shinken/shinken/webui/bottle.py", line 2203, in run
  res = server.run(app)
 File "/usr/local/shinken/shinken/webui/bottle.py", line 2088, in run
  return sa(self.host, self.port, **self.options).run(handler)
 File "/usr/local/shinken/shinken/webui/bottle.py", line 1907, in run
  srv.serve_forever()
 File "/usr/lib/python2.6/SocketServer.py", line 224, in serve_forever
  r, w, e = select.select([self], [], [], poll_interval)
error: (4, 'Interrupted system call')
2012-11-28 11:47:13,152 [1354099633] Error :  Failed sending configuration for receiver-1: receiving: connection lost: [Errno 104] Connection reset by peer
2012-11-28 11:47:13,163 [1354099633] Error :  [All] Dispatching failed for receiver receiver-1
2012-11-28 11:54:30,667 [1354100070] Warning : Printing stored debug messages prior to our daemonization
2012-11-28 11:54:35,224 [1354100075] Warning : Printing stored debug messages prior to our daemonization
2012-11-28 11:54:35,343 [1354100075] Warning : IOError(2, 'No such file or directory')
2012-11-28 12:10:40,502 [1354101040] Critical : I got an unrecoverable error. I have to exit
2012-11-28 12:10:40,504 [1354101040] Critical : You can log a bug ticket at https://github.com/naparuba/shinken/issues/new to get help
2012-11-28 12:10:40,549 [1354101040] Critical : Back trace of it: Traceback (most recent call last):
 File "/usr/local/shinken/shinken/daemons/skonfdaemon.py", line 442, in main
  self.do_mainloop()
 File "/usr/local/shinken/shinken/daemon.py", line 244, in do_mainloop
  self.do_loop_turn()
 File "/usr/local/shinken/shinken/daemons/skonfdaemon.py", line 463, in do_loop_turn
  self.run()
 File "/usr/local/shinken/shinken/daemons/skonfdaemon.py", line 566, in run
  srv = run(host=self.http_host, port=self.http_port, server=self.http_backend)
 File "/usr/local/shinken/shinken/webui/bottle.py", line 2203, in run
  res = server.run(app)
 File "/usr/local/shinken/shinken/webui/bottle.py", line 2088, in run
  return sa(self.host, self.port, **self.options).run(handler)
 File "/usr/local/shinken/shinken/webui/bottle.py", line 1907, in run
  srv.serve_forever()
 File "/usr/lib/python2.6/SocketServer.py", line 224, in serve_forever
  r, w, e = select.select([self], [], [], poll_interval)
error: (4, 'Interrupted system call')
2012-11-28 12:10:43,601 [1354101043] Warning : Printing stored debug messages prior to our daemonization
2012-11-28 12:10:47,267 [1354101047] Warning : Add failed attempt to scheduler-1 (1/3) receiving: not enough data
2012-11-28 12:10:49,371 [1354101049] Warning : Scheduler scheduler-1 did not managed its configuration 0,I am not happy.
2012-11-28 12:10:49,372 [1354101049] Warning : [All] The reactionner reactionner-1 seems to be down, I must re-dispatch its role to someone else.
2012-11-28 12:10:49,372 [1354101049] Warning : [All] The poller poller-1 seems to be down, I must re-dispatch its role to someone else.
2012-11-28 12:10:49,372 [1354101049] Warning : [All] The broker broker-1 seems to be down, I must re-dispatch its role to someone else.
2012-11-28 12:10:49,373 [1354101049] Warning : [All] The receiver receiver-1 seems to be down, I must re-dispatch its role to someone else.
2012-11-28 12:10:49,394 [1354101049] Error :  Failed sending configuration for broker-1: cannot connect: [Errno 111] Connection refused
2012-11-28 12:10:51,580 [1354101051] Warning : Missing satellite broker for configuration 0:
2012-11-28 12:10:51,595 [1354101051] Error :  Failed sending configuration for receiver-1: cannot connect: [Errno 111] Connection refused
2012-11-28 12:10:51,595 [1354101051] Error :  [All] Dispatching failed for receiver receiver-1
2012-11-28 12:10:57,139 [1354101057] Warning : Printing stored debug messages prior to our daemonization
    Logs from poller

    Code:
    2012-11-28 08:58:22,850 [1354089502] Warning : Printing stored debug messages prior to our daemonization
2012-11-28 09:02:49,650 [1354089769] Warning : [poller-1] The worker 0 goes down unexpectly!
2012-11-28 09:02:49,651 [1354089769] Warning : [poller-1] The worker 1 goes down unexpectly!
2012-11-28 09:02:49,724 [1354089769] Warning : [poller-1] Scheduler scheduler-1 is not initialized or has network problem: cannot connect: [Errno 111] Connection refused
2012-11-28 09:02:49,727 [1354089769] Warning : [poller-1] Scheduler scheduler-1 is not initialized or has network problem: cannot connect: [Errno 111] Connection refused
2012-11-28 09:02:49,727 [1354089769] Warning : Sent failed!
2012-11-28 09:04:31,700 [1354089871] Warning : Printing stored debug messages prior to our daemonization
2012-11-28 11:15:46,251 [1354097746] Warning : [poller-1] Scheduler scheduler-1 is not initialized or has network problem: unknown object
2012-11-28 11:15:46,256 [1354097746] Warning : [poller-1] Scheduler scheduler-1 is not initialized or has network problem: unknown object
2012-11-28 11:15:46,257 [1354097746] Warning : Sent failed!
2012-11-28 11:15:47,272 [1354097747] Warning : [poller-1] Scheduler scheduler-1 is not initialized or has network problem: unknown object
2012-11-28 11:15:47,276 [1354097747] Warning : [poller-1] Scheduler scheduler-1 is not initialized or has network problem: unknown object
2012-11-28 11:15:47,277 [1354097747] Warning : Sent failed!
2012-11-28 11:15:48,302 [1354097748] Warning : Printing stored debug messages prior to our daemonization
2012-11-28 11:35:43,718 [1354098943] Warning : [poller-1] Scheduler scheduler-1 is not initialized or has network problem: cannot connect: [Errno 111] Connection refused
2012-11-28 11:35:43,724 [1354098943] Warning : [poller-1] Scheduler scheduler-1 is not initialized or has network problem: cannot connect: [Errno 111] Connection refused
2012-11-28 11:35:43,724 [1354098943] Warning : Sent failed!
2012-11-28 11:35:44,729 [1354098944] Warning : [poller-1] Scheduler scheduler-1 is not initialized or has network problem: unknown object
2012-11-28 11:35:44,731 [1354098944] Warning : [poller-1] Scheduler scheduler-1 is not initialized or has network problem: unknown object
2012-11-28 11:35:44,732 [1354098944] Warning : Sent failed!
2012-11-28 11:35:46,104 [1354098946] Warning : Printing stored debug messages prior to our daemonization
2012-11-28 11:44:28,538 [1354099468] Warning : [poller-1] Scheduler scheduler-1 is not initialized or has network problem: cannot connect: [Errno 111] Connection refused
2012-11-28 11:44:28,540 [1354099468] Warning : [poller-1] Scheduler scheduler-1 is not initialized or has network problem: cannot connect: [Errno 111] Connection refused
2012-11-28 11:44:28,541 [1354099468] Warning : Sent failed!
2012-11-28 11:46:51,439 [1354099611] Warning : Printing stored debug messages prior to our daemonization
2012-11-28 11:47:12,193 [1354099632] Warning : [poller-1] The worker 0 goes down unexpectly!
2012-11-28 11:47:12,194 [1354099632] Warning : [poller-1] The worker 1 goes down unexpectly!
2012-11-28 11:47:12,274 [1354099632] Warning : [poller-1] Scheduler scheduler-1 is not initialized or has network problem: cannot connect: [Errno 111] Connection refused
2012-11-28 11:47:12,276 [1354099632] Warning : [poller-1] Scheduler scheduler-1 is not initialized or has network problem: cannot connect: [Errno 111] Connection refused
2012-11-28 11:47:12,277 [1354099632] Warning : Sent failed!
2012-11-28 11:54:31,698 [1354100071] Warning : Printing stored debug messages prior to our daemonization
2012-11-28 12:10:43,848 [1354101043] Warning : [poller-1] Scheduler scheduler-1 is not initialized or has network problem: cannot connect: [Errno 111] Connection refused
2012-11-28 12:10:43,865 [1354101043] Warning : [poller-1] Scheduler scheduler-1 is not initialized or has network problem: cannot connect: [Errno 111] Connection refused
2012-11-28 12:10:43,865 [1354101043] Warning : Sent failed!
2012-11-28 12:10:44,888 [1354101044] Warning : [poller-1] Scheduler scheduler-1 is not initialized or has network problem: unknown object
2012-11-28 12:10:44,892 [1354101044] Warning : [poller-1] Scheduler scheduler-1 is not initialized or has network problem: unknown object
2012-11-28 12:10:44,893 [1354101044] Warning : Sent failed!
2012-11-28 12:10:45,700 [1354101045] Warning : Printing stored debug messages prior to our daemonization
    logs from brokers (same error but with some warning at the end)
    Code:
    2012-11-28 12:10:02,835 [1354101002] Error :  [broker-1] The instance Graphite-Perfdata raised an exception [Errno 111] Connection refused, I remove it!
2012-11-28 12:10:02,836 [1354101002] Error :  [broker-1] Back trace of this remove: Traceback (most recent call last):
 File "/usr/local/shinken/shinken/modulesmanager.py", line 131, in try_instance_init
  inst.init()
 File "/usr/local/shinken/shinken/modules/graphite_broker.py", line 82, in init
  self.con.connect((self.host, self.port))
 File "<string>", line 1, in connect
error: [Errno 111] Connection refused
2012-11-28 12:10:07,888 [1354101007] Error :  [broker-1] The instance Graphite-Perfdata raised an exception [Errno 111] Connection refused, I remove it!
2012-11-28 12:10:07,889 [1354101007] Error :  [broker-1] Back trace of this remove: Traceback (most recent call last):
 File "/usr/local/shinken/shinken/modulesmanager.py", line 131, in try_instance_init
  inst.init()
 File "/usr/local/shinken/shinken/modules/graphite_broker.py", line 82, in init
  self.con.connect((self.host, self.port))
 File "<string>", line 1, in connect
error: [Errno 111] Connection refused
2012-11-28 12:10:44,229 [1354101044] Warning : [broker-1] Connection problem to the scheduler scheduler-1: receiving: not enough data
2012-11-28 12:10:45,245 [1354101045] Warning : [broker-1] Connection problem to the poller poller-1: receiving: not enough data
2012-11-28 12:10:46,255 [1354101046] Warning : [broker-1] Connection problem to the reactionner reactionner-1: receiving: not enough data
2012-11-28 12:10:50,002 [1354101050] Warning : Printing stored debug messages prior to our daemonization

    someone understand that ? ???
    Reply With Quote Reply With Quote
  9. 11-28-2012, 01:49 PM #9
    xkilian
    xkilian is offline
    Administrator
    Join Date
    Dec 2011
    Posts
    278

    Re: Shinken/Graphite on separate vmware esxi5 server [Errno 111] Connection refused

    Hi Piellick,

    Okay, you are getting connection refused problems. Please run a trace with the command and attach the trace to your reply. Not with a cut and paste. This way I can open it in Wireshark. Let the trace run for a while until you get a connection refused problem in your broker log. :-)

    tcpdump host 192.168.1.130 port 2003 -w packet_trace.pcap

    You have a problem with the communications between the two hosts. Have you looked in the /var/log/messages on the Graphite server?

    Because it happens on many different ports and processes, this is IMO not a Shinken issue.

    Cheers,

    xkilian
    Reply With Quote Reply With Quote
  10. 11-29-2012, 11:33 AM #10
    piellick
    piellick is offline
    Member
    Join Date
    Jun 2012
    Posts
    36

    Re: Shinken/Graphite on separate vmware esxi5 server [Errno 111] Connection refused

    Hi everyone,

    sorry for the Cut and Paste ;D there is ....

    my shinken broker logs (on debug mode) : http://s440093767.onlinehome.fr/debu...bug_broker.txt

    A tcp dump with 200 packet from shinken VM to Graphite (connection looks good ???) create with tcpdump -nnvvXSs 1514 host 192.168.1.130 -c 200 -w shinken.pcap
    http://s440093767.onlinehome.fr/debu...n/shinken.pcap

    a tcp dump with 200 packet from graphite VM create with tcpdump -nnvvXSs 1514 host 192.168.1.71 -c 200 -w grpahite.pcap

    http://s440093767.onlinehome.fr/debu.../graphite.pcap

    Graphite logs :

    Webapp exception.log
    http://s440093767.onlinehome.fr/debug_shinken/logs_graphite/webapp/exception.log

    Webapp info.log
    http://s440093767.onlinehome.fr/debu...ebapp/info.log

    carbon cache log
    http://s440093767.onlinehome.fr/debu...-a/console.log
    http://s440093767.onlinehome.fr/debu...-a/creates.log
    http://s440093767.onlinehome.fr/debu...he-a/query.log
    http://s440093767.onlinehome.fr/debu...a/listener.log

    I hope we find a solution... ;D

    Reply With Quote Reply With Quote

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

Forum Rules