Results 1 to 10 of 10

Thread: Creates new eth0_XXX graphs at each poll - disk full

  1. #1
    Junior Member
    Join Date
    Apr 2013
    Posts
    20

    Creates new eth0_XXX graphs at each poll - disk full

    Hi,

    Graphite is creating new network graphs at each poll, which quickly fills up the disk:
    _eth0_in_discard_XXX_.wsp
    _eth0_in_error_XXX_.wsp
    _eth0_in_octet_XXX_.wsp
    _eth0_out_discard_XXX_.wsp
    _eth0_out_error_XXX_.wsp
    _eth0_out_octet_XXX_.wsp
    where XXX is the current timestamp (e.g. 1368111520).

    This sounds like a bug (Note: I have Shinken 1.2.4 here).
    Is there a way to prevent this? :-\

  2. #2
    Shinken project leader
    Join Date
    May 2011
    Location
    Bordeaux (France)
    Posts
    2,131

    Re: Creates new eth0_XXX graphs at each poll - disk full

    Can you try to open in debug mode your broker? We will see the packet sent, because I don't see such thing in the code.
    No direct support by personal message. Please open a thread so everyone can see the solution

  3. #3
    Junior Member
    Join Date
    Apr 2013
    Posts
    20

    Re: Creates new eth0_XXX graphs at each poll - disk full

    For info here are 2 sets of performance data for localhost:
    Code:
    'eth0_in_octet'=2089342344c 'eth0_out_octet'=2756168837c 'eth0_in_error'=0 'eth0_out_error'=0 'eth0_in_discard'=0 'eth0_out_discard'=0 cache_descr_ids=2 cache_descr_names=eth0 cache_descr_time=1368147520 'eth0_in_octet.1368176921'=2088692800 'eth0_out_octet.1368176921'=2755753709 'eth0_in_error.1368176921'=0 'eth0_out_error.1368176921'=0 'eth0_in_discard.1368176921'=0 'eth0_out_discard.1368176921'=0 ptime=1368177220 
    
    'eth0_in_octet'=2090660687c 'eth0_out_octet'=2756997879c 'eth0_in_error'=0 'eth0_out_error'=0 'eth0_in_discard'=0 'eth0_out_discard'=0 cache_descr_ids=2 cache_descr_names=eth0 cache_descr_time=1368147520 'eth0_in_octet.1368177520'=2090008849 'eth0_out_octet.1368177520'=2756624763 'eth0_in_error.1368177520'=0 'eth0_out_error.1368177520'=0 'eth0_in_discard.1368177520'=0 'eth0_out_discard.1368177520'=0 ptime=1368177820
    Note the "eth0_in_error.1368176921=..." kind of keys.

    It seems related to the "-P"&"-T" option in the ./check_netint.pl" commands:
    Code:
    # ./check_netint.pl -H localhost -C public -n "eth\d+|em\d+" -f -e -w 90,90,0,0,0,0 -c 0,0,0,0,0,0 -q -k -y -M -B -m -P "'eth0_in_error'=0 'eth0_out_error'=0 'eth0_in_discard'=0 'eth0_out_discard'=0"
    eth0:UP (no usable data - 1 rows) (1 UP): OK | 'eth0_in_octet'=2095739589c 'eth0_out_octet'=2760380718c 'eth0_in_error'=0 'eth0_out_error'=0 'eth0_in_discard'=0 'eth0_out_discard'=0 cache_descr_ids=2 cache_descr_names=eth0 cache_descr_time=1368179768 ptime=1368179768
    
    # ./check_netint.pl -H localhost -C public -n "eth\d+|em\d+" -f -e -w 90,90,0,0,0,0 -c 0,0,0,0,0,0 -q -k -y -M -B -m -P "'eth0_in_error'=0 'eth0_out_error'=0 'eth0_in_discard'=0 'eth0_out_discard'=0" -T $(date -d 'now - 5 minutes' +%s)
    eth0:UP (no usable data - 1 rows) (1 UP): OK | 'eth0_in_octet'=2095771493c 'eth0_out_octet'=2760534332c 'eth0_in_error'=0 'eth0_out_error'=0 'eth0_in_discard'=0 'eth0_out_discard'=0 cache_descr_ids=2 cache_descr_names=eth0 cache_descr_time=1368179862 'eth0_in_error.1368179562'=0 'eth0_out_error.1368179562'=0 'eth0_in_discard.1368179562'=0 'eth0_out_discard.1368179562'=0 ptime=1368179862

  4. #4
    Shinken project leader
    Join Date
    May 2011
    Location
    Bordeaux (France)
    Posts
    2,131

    Re: Creates new eth0_XXX graphs at each poll - disk full

    Arg, maybe linked to the LASTPERFDATA thing, but it's not good at all this thing
    No direct support by personal message. Please open a thread so everyone can see the solution

  5. #5
    Junior Member
    Join Date
    Apr 2013
    Posts
    20

    Re: Creates new eth0_XXX graphs at each poll - disk full

    AFAICS you can't ignore these graphs in Graphite's storage-schemas.conf, only mitigate the problem by using tiny archives :
    Code:
    [shinken_ignore_dummy_network]
    pattern = .*\.shinken\.NetworkUsage\..*_[0-9]{10}_
    retentions = 0:0
    so I guess we need to fix [tt]check_netint.pl[/tt] so it doesn't output these keys. :

  6. #6
    Shinken project leader
    Join Date
    May 2011
    Location
    Bordeaux (France)
    Posts
    2,131

    Re: Creates new eth0_XXX graphs at each poll - disk full

    Yes, or use a constant name for metric name, because here it's too dangerous
    No direct support by personal message. Please open a thread so everyone can see the solution

  7. #7
    Junior Member
    Join Date
    Oct 2012
    Posts
    6

    Re: Creates new eth0_XXX graphs at each poll - disk full

    Hello,

    I have the same pb with Shinken 1.4 and check_netint.pl v2.4a9, even with the filter added in storage-schemas.conf (data are still there but with a rentention 0).

    What could we do to remove all these bad entries in graphite db and use this powerfull plugin ?

    Thor.

  8. #8
    Shinken project leader
    Join Date
    May 2011
    Location
    Bordeaux (France)
    Posts
    2,131

    Re: Creates new eth0_XXX graphs at each poll - disk full

    In graphite you can just go to the wisper directory and delete the wsp files.
    No direct support by personal message. Please open a thread so everyone can see the solution

  9. #9
    Junior Member
    Join Date
    Oct 2012
    Posts
    6

    Re: Creates new eth0_XXX graphs at each poll - disk full

    Hello,

    A better solution (before having a new check_netint.pl version) would be to use the blacklist/whitelist graphite process.

    1/ in carbon.conf file, section [cache]:
    add USE_WHITELIST = True

    2/ copy blacklist.conf.example in blacklist.conf

    3/ add regular expression to skip eth0_in_octet and eth0_out_octet measures.

    Thor.

  10. #10
    Shinken project leader
    Join Date
    May 2011
    Location
    Bordeaux (France)
    Posts
    2,131

    Re: Creates new eth0_XXX graphs at each poll - disk full

    Oh great idea
    No direct support by personal message. Please open a thread so everyone can see the solution

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •