I currently have a setup with a few mod_gearman workers with Icinga on the main box. We have a few zones where the workers can contact the main VM with the work queue's, the main site/realm/box with all the config is not allowed for security reasons to speak to the workers.

Is this also possible in Shinken? Set up the site to use a TCP session to fetch everything and use that session to retreive data and push results.