Page 1 of 2 12 LastLast
Results 1 to 10 of 12

Thread: Slow response searching events

  1. #1
    Senior Member
    Join Date
    Aug 2011
    Posts
    201

    Slow response searching events

    Hi
    I've my installation with (about) 1000 hosts and 300 services.
    A lot of hosts are routers or switches with a simple ping check (about 650).

    I've a problem when searching for a incident from multisite UI. The search is very slow and shinken hosts gets stuck for a while..
    There is any way to search log "friendly" from multisite? or which method can I use to do it? any suggestion?

    thanks!

  2. #2
    Shinken project leader
    Join Date
    May 2011
    Location
    Bordeaux (France)
    Posts
    2,131

    Re: Slow response searching events

    Is it a 1.0.1 version? (version before was using one huge sqlite database, and so was very slow).

    For 1.0.1 and the current git version, there should be one db by day, and so should be quicker for last days search (if you search for the last months, it will browse all db, and so will be slow).

    One way to get better performances here is to switch to the mongodb log backend. There is a sample of such module in the shinken-specific.cfg file (search log + mongodb). When linked to the LiveStatus module, it will output logs into the mongodb database, that is quite faster than sqlite
    No direct support by personal message. Please open a thread so everyone can see the solution

  3. #3
    Senior Member
    Join Date
    Aug 2011
    Posts
    201

    Re: Slow response searching events

    Hi naparuba
    I'm using 1.0.1. I reviewed shinken-specific.cfg to look for used modules.
    I'm currently using
    broker -->Livestatus, Simple-log, WebUI,PickleRetentionBroker,NPCDMOD
    and from Livestatus-->logsqlite (and tryied mongodb at other test shinken server).

    I have simple-log (I thinkg this is the slow ) because I thought that it was mandatory for accesing logs from Multisite Ui.. it is correct? Can I erase it and livestatus will use sqlite or mongodb events?

    I'm a bit confused with this

    thanks

  4. #4
    Administrator
    Join Date
    Dec 2011
    Posts
    278

    Re: Slow response searching events

    Multisite only uses Livestatus->logsqlite or Livestatus->mongodb

    Simple-log is used for logging for the of the Shinken daemons. It has nothing to do with Livestatus.

    If you are using a very large configuration the fastest is:

    Livestatus->logsqlite

    Add this line to your sqlite logstore module configuration, this will optimize queries to use sql when it can and not in-memory structures :
    use_aggressive_sql 1

    This option does not exist for mongodb logstore.I also noticed that the sample configuration does not have the option listed. Need to fix the doc. :-)

    If you are running more than 10K services you absolutely need to run the git version and not stock 1.0.1.

    Cheers,

    X

  5. #5
    Administrator
    Join Date
    Dec 2011
    Posts
    278

    Re: Slow response searching events

    Fixed the wiki.
    Now I have to modify the sample shinken-specific.cfg files to mention the existence of the option.

    How many services are you testing.
    What types of queries are slow?

    The multisite web 2.0 search bar is a piece of crap which slows everything down, we replaced it with a regex based search. We have not published the change. I will see if I can dig it up.
    Nagvis also slows down the webui, because the the little coloured dot means it is actually querying the state of ALL the hosts and services that form each dot. eck.

    Have you checked the cpu on the host running the broker?
    Do you have errors in your broker log file?

    Cheers,

    X

  6. #6
    Senior Member
    Join Date
    Aug 2011
    Posts
    201

    Re: Slow response searching events

    Hi xkilian
    At broker.log there are no relevant errors... some warnings about not used modules at restart services. No errors as I can see.

    Problems come when I look for events or notifications for some time period at Multisite (using 1.1.12p7). I've limited these parameters at multisite.mk for cutting query
    soft_query_limit = 1000
    hard_query_limit = 2000

    when I search server's cpu goes up to 88 or 90% cpu usage (normal working cpu usage is <10%) and shinken user, python process has this high cpu usage (so, shinken-livestatus is searching???)

    at shinken/var/archives I've a lot of files named "nagios-mm-dd-yyyy.log" (7megas or 23megas) and "livelogs-yyyy-mm-dd.db" and .db-journal, these are sqlite files,?

    searches remains very slow...

    do you know other method (not multisite) to search these logs files for some event (date, host or service search)?? I don't love specially multisite, we only use it for operator's videowall.... just need to search incidents but can use other tool to do it and let multisite only for viewing

    thank you a lot


  7. #7
    Administrator
    Join Date
    Dec 2011
    Posts
    278

    Re: Slow response searching events

    Hmm..

    Hmmm.. When Livestatus is busy, it is normal for the cpu to spkie. It cam only access a single cpu core at a time. Thank you Python. :-/ But queries against SQLite are threaded and use_aggressive_sql should help for smaller queries and to use SQL indexes instead of python lists.

    You are using SQLite3? You can rty putting the files up in a RAMisk, simply for speed.
    For event analysis, profiliing, SLA calculation, ets. When are planning to simply export all event and log broks to Splunk. Splunk has the facilities to index the data correctly, split the load, build custom dashboards, searches for SLA reports. It can then feedback events into Shinken using send_nsca.

    Event reporting in Shinken or Multisite or Thruk, etc. Should focus on common use and easy wins. SLA reports would fit here, Top N, etc. But these are open-source with small commercial shops, and reporting is not easy. So I just use a shortcut to meet $DAYJOB$ requirements. Will keep you posted on how the Splunk integration goes.

    Otherwise, you could run Jasper reports against Mongodb, but Mongodb support with Shinken does not support use_aggressive_sql yet and still has missing/broken features. Once Mongodb is there, I could certainly envision, querying it directly for data and bypassing LS for some queries using Jasper or even Multisite or Nagvis.

    --- back to problem

    The query limits will not stop LS to look through everything in the DBs until it fills up the N elements (1000).
    You need to enable debug of sqlite to log queries and look at what exactly it is running as a query. To find, is it SQLite that is slow, or LS processing before or after the query.

    Titilambert or Gerhard Lausser would be the right people to provide you more detailed feedback. You need to dig a bit.

    Cheers.

    X

  8. #8
    Senior Member
    Join Date
    Aug 2011
    Posts
    201

    Re: Slow response searching events

    thanks a lot xkilian

    i will read slow and try

  9. #9
    Administrator
    Join Date
    Dec 2011
    Posts
    278

    Re: Slow response searching events

    In your debugging path you can also enable Livestatus debug, it is an option to the module. This will help you profile where it is spending its time.

    Cheers,

    X

  10. #10
    Administrator
    Join Date
    Dec 2011
    Posts
    278

    Re: Slow response searching events

    An open issue #462 might explain why it is so slow.

    There is a function missing in Livestatus to filter logs and there may be other optimizations necessary for fast SQL like requests via Livestatus against the various log types.

    This will improve memory usage and database file size.

    Once the fix is in, you can try it again.

    For those not following commits:
    • [li]vaxvms put in a fix for the long running mongodb logstore bug where some queries types wher returning empty[/li]
      [li]olivierha put in a new null logstore for Nagvis dedicated Livestatus brokers, more details to come[/li]
      [li]olivierha put in a fix to speed up event queries by limiting the timespan, this will help mongodb users and sqlite users without the use_aggressive_sql option[/li]


    Cheers,

    xkilian

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •