The IceProd framework

Distributed data processing for the IceCube neutrino observatory

M. G. Aartsen, R. Abbasi, M. Ackermann, J. Adams, J. A. Aguilar, M. Ahlers, D. Altmann, C. Arguelles, J. Auffenberg, X. Bai, M. Baker, S. W. Barwick, V. Baum, R. Bay, J. J. Beatty, J. Becker Tjus, K. H. Becker, S. Benzvi, P. Berghaus, D. Berley & 267 others E. Bernardini, A. Bernhard, D. Z. Besson, G. Binder, D. Bindig, M. Bissok, E. Blaufuss, J. Blumenthal, D. J. Boersma, C. Bohm, D. Bose, S. Böser, O. Botner, L. Brayeur, H. P. Bretz, A. M. Brown, R. Bruijn, J. Casey, M. Casier, D. Chirkin, A. Christov, B. Christy, K. Clark, L. Classen, F. Clevermann, S. Coenders, S. Cohen, Douglas Cowen, A. H. Cruz Silva, M. Danninger, J. Daughhetee, J. C. Davis, M. Day, C. De Clercq, S. De Ridder, P. Desiati, K. D. De Vries, M. De With, T. Deyoung, J. C. Díaz-Vélez, M. Dunkman, R. Eagan, B. Eberhardt, B. Eichmann, J. Eisch, S. Euler, P. A. Evenson, O. Fadiran, A. R. Fazely, A. Fedynitch, J. Feintzeig, T. Feusels, K. Filimonov, C. Finley, T. Fischer-Wasels, S. Flis, A. Franckowiak, K. Frantzen, T. Fuchs, T. K. Gaisser, J. Gallagher, L. Gerhardt, L. Gladstone, T. Glüsenkamp, A. Goldschmidt, G. Golup, J. G. Gonzalez, J. A. Goodman, D. Góra, D. T. Grandmont, D. Grant, P. Gretskov, J. C. Groh, A. Groß, C. Ha, A. Haj Ismail, P. Hallen, A. Hallgren, F. Halzen, K. Hanson, D. Hebecker, D. Heereman, D. Heinen, K. Helbing, R. Hellauer, S. Hickford, G. C. Hill, K. D. Hoffman, R. Hoffmann, A. Homeier, K. Hoshina, F. Huang, W. Huelsnitz, P. O. Hulth, K. Hultqvist, S. Hussain, A. Ishihara, E. Jacobi, J. Jacobsen, K. Jagielski, G. S. Japaridze, K. Jero, O. Jlelati, B. Kaminsky, A. Kappes, T. Karg, A. Karle, M. Kauer, J. L. Kelley, J. Kiryluk, J. Kläs, S. R. Klein, J. H. Köhne, G. Kohnen, H. Kolanoski, L. Köpke, C. Kopper, S. Kopper, D. J. Koskinen, M. Kowalski, M. Krasberg, A. Kriesten, K. Krings, G. Kroll, J. Kunnen, N. Kurahashi, T. Kuwabara, M. Labare, H. Landsman, M. J. Larson, M. Lesiak-Bzdak, M. Leuermann, J. Leute, J. Lünemann, O. Macías, J. Madsen, G. Maggi, R. Maruyama, K. Mase, H. S. Matis, F. McNally, K. Meagher, M. Merck, G. Merino, T. Meures, S. Miarecki, E. Middell, N. Milke, J. Miller, L. Mohrmann, T. Montaruli, R. Morse, R. Nahnhauer, U. Naumann, H. Niederhausen, S. C. Nowicki, D. R. Nygren, A. Obertacke, S. Odrowski, A. Olivas, A. Omairat, A. O'Murchadha, L. Paul, J. A. Pepper, C. Pérez De Los Heros, C. Pfendner, D. Pieloth, E. Pinat, J. Posselt, P. B. Price, G. T. Przybylski, M. Quinnan, L. Rädel, I. Rae, M. Rameez, K. Rawlins, P. Redl, R. Reimann, E. Resconi, W. Rhode, M. Ribordy, M. Richman, B. Riedel, J. P. Rodrigues, C. Rott, T. Ruhe, B. Ruzybayev, D. Ryckbosch, S. M. Saba, H. G. Sander, M. Santander, S. Sarkar, K. Schatto, F. Scheriau, T. Schmidt, M. Schmitz, S. Schoenen, S. Schöneberg, A. Schönwald, A. Schukraft, L. Schulte, D. Schultz, O. Schulz, D. Seckel, Y. Sestayo, S. Seunarine, R. Shanidze, C. Sheremata, M. W.E. Smith, D. Soldin, G. M. Spiczak, C. Spiering, M. Stamatikos, T. Stanev, N. A. Stanisha, A. Stasik, T. Stezelberger, R. G. Stokstad, A. Stößl, E. A. Strahler, R. Ström, N. L. Strotjohann, G. W. Sullivan, H. Taavola, I. Taboada, A. Tamburro, A. Tepe, S. Ter-Antonyan, G. Tešić, S. Tilav, P. A. Toale, M. N. Tobin, S. Toscano, M. Tselengidou, E. Unger, M. Usner, S. Vallecorsa, N. Van Eijndhoven, A. Van Overloop, J. Van Santen, M. Vehring, M. Voge, M. Vraeghe, C. Walck, T. Waldenmaier, M. Wallraff, Ch Weaver, M. Wellons, C. Wendt, S. Westerhoff, N. Whitehorn, K. Wiebe, C. H. Wiebusch, D. R. Williams, H. Wissing, M. Wolf, T. R. Wood, K. Woschnagg, D. L. Xu, X. W. Xu, J. P. Yanez, G. Yodh, S. Yoshida, P. Zarzhitsky, J. Ziemann, S. Zierke, M. Zoll

Research output: Contribution to journalArticle

4 Citations (Scopus)

Abstract

IceCube is a one-gigaton instrument located at the geographic South Pole, designed to detect cosmic neutrinos, identify the particle nature of dark matter, and study high-energy neutrinos themselves. Simulation of the IceCube detector and processing of data require a significant amount of computational resources. This paper presents the first detailed description of IceProd, a lightweight distributed management system designed to meet these requirements. It is driven by a central database in order to manage mass production of simulations and analysis of data produced by the IceCube detector. IceProd runs as a separate layer on top of other middleware and can take advantage of a variety of computing resources, including grids and batch systems such as CREAM, HTCondor, and PBS. This is accomplished by a set of dedicated daemons that process job submission in a coordinated fashion through the use of middleware plugins that serve to abstract the details of job submission and job management from the framework.

Original languageEnglish (US)
Pages (from-to)198-211
Number of pages14
JournalJournal of Parallel and Distributed Computing
Volume75
DOIs
StatePublished - Jan 1 2015

Fingerprint

Distributed Processing
Observatories
Middleware
Observatory
Neutrinos
Detector
Detectors
Resources
Dark Matter
Batch
High Energy
Pole
Poles
Simulation
Grid
Computing
Requirements
Processing
Framework

All Science Journal Classification (ASJC) codes

  • Software
  • Theoretical Computer Science
  • Hardware and Architecture
  • Computer Networks and Communications
  • Artificial Intelligence

Cite this

Aartsen, M. G., Abbasi, R., Ackermann, M., Adams, J., Aguilar, J. A., Ahlers, M., ... Zoll, M. (2015). The IceProd framework: Distributed data processing for the IceCube neutrino observatory. Journal of Parallel and Distributed Computing, 75, 198-211. https://doi.org/10.1016/j.jpdc.2014.08.001
Aartsen, M. G. ; Abbasi, R. ; Ackermann, M. ; Adams, J. ; Aguilar, J. A. ; Ahlers, M. ; Altmann, D. ; Arguelles, C. ; Auffenberg, J. ; Bai, X. ; Baker, M. ; Barwick, S. W. ; Baum, V. ; Bay, R. ; Beatty, J. J. ; Becker Tjus, J. ; Becker, K. H. ; Benzvi, S. ; Berghaus, P. ; Berley, D. ; Bernardini, E. ; Bernhard, A. ; Besson, D. Z. ; Binder, G. ; Bindig, D. ; Bissok, M. ; Blaufuss, E. ; Blumenthal, J. ; Boersma, D. J. ; Bohm, C. ; Bose, D. ; Böser, S. ; Botner, O. ; Brayeur, L. ; Bretz, H. P. ; Brown, A. M. ; Bruijn, R. ; Casey, J. ; Casier, M. ; Chirkin, D. ; Christov, A. ; Christy, B. ; Clark, K. ; Classen, L. ; Clevermann, F. ; Coenders, S. ; Cohen, S. ; Cowen, Douglas ; Cruz Silva, A. H. ; Danninger, M. ; Daughhetee, J. ; Davis, J. C. ; Day, M. ; De Clercq, C. ; De Ridder, S. ; Desiati, P. ; De Vries, K. D. ; De With, M. ; Deyoung, T. ; Díaz-Vélez, J. C. ; Dunkman, M. ; Eagan, R. ; Eberhardt, B. ; Eichmann, B. ; Eisch, J. ; Euler, S. ; Evenson, P. A. ; Fadiran, O. ; Fazely, A. R. ; Fedynitch, A. ; Feintzeig, J. ; Feusels, T. ; Filimonov, K. ; Finley, C. ; Fischer-Wasels, T. ; Flis, S. ; Franckowiak, A. ; Frantzen, K. ; Fuchs, T. ; Gaisser, T. K. ; Gallagher, J. ; Gerhardt, L. ; Gladstone, L. ; Glüsenkamp, T. ; Goldschmidt, A. ; Golup, G. ; Gonzalez, J. G. ; Goodman, J. A. ; Góra, D. ; Grandmont, D. T. ; Grant, D. ; Gretskov, P. ; Groh, J. C. ; Groß, A. ; Ha, C. ; Haj Ismail, A. ; Hallen, P. ; Hallgren, A. ; Halzen, F. ; Hanson, K. ; Hebecker, D. ; Heereman, D. ; Heinen, D. ; Helbing, K. ; Hellauer, R. ; Hickford, S. ; Hill, G. C. ; Hoffman, K. D. ; Hoffmann, R. ; Homeier, A. ; Hoshina, K. ; Huang, F. ; Huelsnitz, W. ; Hulth, P. O. ; Hultqvist, K. ; Hussain, S. ; Ishihara, A. ; Jacobi, E. ; Jacobsen, J. ; Jagielski, K. ; Japaridze, G. S. ; Jero, K. ; Jlelati, O. ; Kaminsky, B. ; Kappes, A. ; Karg, T. ; Karle, A. ; Kauer, M. ; Kelley, J. L. ; Kiryluk, J. ; Kläs, J. ; Klein, S. R. ; Köhne, J. H. ; Kohnen, G. ; Kolanoski, H. ; Köpke, L. ; Kopper, C. ; Kopper, S. ; Koskinen, D. J. ; Kowalski, M. ; Krasberg, M. ; Kriesten, A. ; Krings, K. ; Kroll, G. ; Kunnen, J. ; Kurahashi, N. ; Kuwabara, T. ; Labare, M. ; Landsman, H. ; Larson, M. J. ; Lesiak-Bzdak, M. ; Leuermann, M. ; Leute, J. ; Lünemann, J. ; Macías, O. ; Madsen, J. ; Maggi, G. ; Maruyama, R. ; Mase, K. ; Matis, H. S. ; McNally, F. ; Meagher, K. ; Merck, M. ; Merino, G. ; Meures, T. ; Miarecki, S. ; Middell, E. ; Milke, N. ; Miller, J. ; Mohrmann, L. ; Montaruli, T. ; Morse, R. ; Nahnhauer, R. ; Naumann, U. ; Niederhausen, H. ; Nowicki, S. C. ; Nygren, D. R. ; Obertacke, A. ; Odrowski, S. ; Olivas, A. ; Omairat, A. ; O'Murchadha, A. ; Paul, L. ; Pepper, J. A. ; Pérez De Los Heros, C. ; Pfendner, C. ; Pieloth, D. ; Pinat, E. ; Posselt, J. ; Price, P. B. ; Przybylski, G. T. ; Quinnan, M. ; Rädel, L. ; Rae, I. ; Rameez, M. ; Rawlins, K. ; Redl, P. ; Reimann, R. ; Resconi, E. ; Rhode, W. ; Ribordy, M. ; Richman, M. ; Riedel, B. ; Rodrigues, J. P. ; Rott, C. ; Ruhe, T. ; Ruzybayev, B. ; Ryckbosch, D. ; Saba, S. M. ; Sander, H. G. ; Santander, M. ; Sarkar, S. ; Schatto, K. ; Scheriau, F. ; Schmidt, T. ; Schmitz, M. ; Schoenen, S. ; Schöneberg, S. ; Schönwald, A. ; Schukraft, A. ; Schulte, L. ; Schultz, D. ; Schulz, O. ; Seckel, D. ; Sestayo, Y. ; Seunarine, S. ; Shanidze, R. ; Sheremata, C. ; Smith, M. W.E. ; Soldin, D. ; Spiczak, G. M. ; Spiering, C. ; Stamatikos, M. ; Stanev, T. ; Stanisha, N. A. ; Stasik, A. ; Stezelberger, T. ; Stokstad, R. G. ; Stößl, A. ; Strahler, E. A. ; Ström, R. ; Strotjohann, N. L. ; Sullivan, G. W. ; Taavola, H. ; Taboada, I. ; Tamburro, A. ; Tepe, A. ; Ter-Antonyan, S. ; Tešić, G. ; Tilav, S. ; Toale, P. A. ; Tobin, M. N. ; Toscano, S. ; Tselengidou, M. ; Unger, E. ; Usner, M. ; Vallecorsa, S. ; Van Eijndhoven, N. ; Van Overloop, A. ; Van Santen, J. ; Vehring, M. ; Voge, M. ; Vraeghe, M. ; Walck, C. ; Waldenmaier, T. ; Wallraff, M. ; Weaver, Ch ; Wellons, M. ; Wendt, C. ; Westerhoff, S. ; Whitehorn, N. ; Wiebe, K. ; Wiebusch, C. H. ; Williams, D. R. ; Wissing, H. ; Wolf, M. ; Wood, T. R. ; Woschnagg, K. ; Xu, D. L. ; Xu, X. W. ; Yanez, J. P. ; Yodh, G. ; Yoshida, S. ; Zarzhitsky, P. ; Ziemann, J. ; Zierke, S. ; Zoll, M. / The IceProd framework : Distributed data processing for the IceCube neutrino observatory. In: Journal of Parallel and Distributed Computing. 2015 ; Vol. 75. pp. 198-211.
@article{4bde52d471d247158fb80fa49b98fcbc,
title = "The IceProd framework: Distributed data processing for the IceCube neutrino observatory",
abstract = "IceCube is a one-gigaton instrument located at the geographic South Pole, designed to detect cosmic neutrinos, identify the particle nature of dark matter, and study high-energy neutrinos themselves. Simulation of the IceCube detector and processing of data require a significant amount of computational resources. This paper presents the first detailed description of IceProd, a lightweight distributed management system designed to meet these requirements. It is driven by a central database in order to manage mass production of simulations and analysis of data produced by the IceCube detector. IceProd runs as a separate layer on top of other middleware and can take advantage of a variety of computing resources, including grids and batch systems such as CREAM, HTCondor, and PBS. This is accomplished by a set of dedicated daemons that process job submission in a coordinated fashion through the use of middleware plugins that serve to abstract the details of job submission and job management from the framework.",
author = "Aartsen, {M. G.} and R. Abbasi and M. Ackermann and J. Adams and Aguilar, {J. A.} and M. Ahlers and D. Altmann and C. Arguelles and J. Auffenberg and X. Bai and M. Baker and Barwick, {S. W.} and V. Baum and R. Bay and Beatty, {J. J.} and {Becker Tjus}, J. and Becker, {K. H.} and S. Benzvi and P. Berghaus and D. Berley and E. Bernardini and A. Bernhard and Besson, {D. Z.} and G. Binder and D. Bindig and M. Bissok and E. Blaufuss and J. Blumenthal and Boersma, {D. J.} and C. Bohm and D. Bose and S. B{\"o}ser and O. Botner and L. Brayeur and Bretz, {H. P.} and Brown, {A. M.} and R. Bruijn and J. Casey and M. Casier and D. Chirkin and A. Christov and B. Christy and K. Clark and L. Classen and F. Clevermann and S. Coenders and S. Cohen and Douglas Cowen and {Cruz Silva}, {A. H.} and M. Danninger and J. Daughhetee and Davis, {J. C.} and M. Day and {De Clercq}, C. and {De Ridder}, S. and P. Desiati and {De Vries}, {K. D.} and {De With}, M. and T. Deyoung and D{\'i}az-V{\'e}lez, {J. C.} and M. Dunkman and R. Eagan and B. Eberhardt and B. Eichmann and J. Eisch and S. Euler and Evenson, {P. A.} and O. Fadiran and Fazely, {A. R.} and A. Fedynitch and J. Feintzeig and T. Feusels and K. Filimonov and C. Finley and T. Fischer-Wasels and S. Flis and A. Franckowiak and K. Frantzen and T. Fuchs and Gaisser, {T. K.} and J. Gallagher and L. Gerhardt and L. Gladstone and T. Gl{\"u}senkamp and A. Goldschmidt and G. Golup and Gonzalez, {J. G.} and Goodman, {J. A.} and D. G{\'o}ra and Grandmont, {D. T.} and D. Grant and P. Gretskov and Groh, {J. C.} and A. Gro{\ss} and C. Ha and {Haj Ismail}, A. and P. Hallen and A. Hallgren and F. Halzen and K. Hanson and D. Hebecker and D. Heereman and D. Heinen and K. Helbing and R. Hellauer and S. Hickford and Hill, {G. C.} and Hoffman, {K. D.} and R. Hoffmann and A. Homeier and K. Hoshina and F. Huang and W. Huelsnitz and Hulth, {P. O.} and K. Hultqvist and S. Hussain and A. Ishihara and E. Jacobi and J. Jacobsen and K. Jagielski and Japaridze, {G. S.} and K. Jero and O. Jlelati and B. Kaminsky and A. Kappes and T. Karg and A. Karle and M. Kauer and Kelley, {J. L.} and J. Kiryluk and J. Kl{\"a}s and Klein, {S. R.} and K{\"o}hne, {J. H.} and G. Kohnen and H. Kolanoski and L. K{\"o}pke and C. Kopper and S. Kopper and Koskinen, {D. J.} and M. Kowalski and M. Krasberg and A. Kriesten and K. Krings and G. Kroll and J. Kunnen and N. Kurahashi and T. Kuwabara and M. Labare and H. Landsman and Larson, {M. J.} and M. Lesiak-Bzdak and M. Leuermann and J. Leute and J. L{\"u}nemann and O. Mac{\'i}as and J. Madsen and G. Maggi and R. Maruyama and K. Mase and Matis, {H. S.} and F. McNally and K. Meagher and M. Merck and G. Merino and T. Meures and S. Miarecki and E. Middell and N. Milke and J. Miller and L. Mohrmann and T. Montaruli and R. Morse and R. Nahnhauer and U. Naumann and H. Niederhausen and Nowicki, {S. C.} and Nygren, {D. R.} and A. Obertacke and S. Odrowski and A. Olivas and A. Omairat and A. O'Murchadha and L. Paul and Pepper, {J. A.} and {P{\'e}rez De Los Heros}, C. and C. Pfendner and D. Pieloth and E. Pinat and J. Posselt and Price, {P. B.} and Przybylski, {G. T.} and M. Quinnan and L. R{\"a}del and I. Rae and M. Rameez and K. Rawlins and P. Redl and R. Reimann and E. Resconi and W. Rhode and M. Ribordy and M. Richman and B. Riedel and Rodrigues, {J. P.} and C. Rott and T. Ruhe and B. Ruzybayev and D. Ryckbosch and Saba, {S. M.} and Sander, {H. G.} and M. Santander and S. Sarkar and K. Schatto and F. Scheriau and T. Schmidt and M. Schmitz and S. Schoenen and S. Sch{\"o}neberg and A. Sch{\"o}nwald and A. Schukraft and L. Schulte and D. Schultz and O. Schulz and D. Seckel and Y. Sestayo and S. Seunarine and R. Shanidze and C. Sheremata and Smith, {M. W.E.} and D. Soldin and Spiczak, {G. M.} and C. Spiering and M. Stamatikos and T. Stanev and Stanisha, {N. A.} and A. Stasik and T. Stezelberger and Stokstad, {R. G.} and A. St{\"o}{\ss}l and Strahler, {E. A.} and R. Str{\"o}m and Strotjohann, {N. L.} and Sullivan, {G. W.} and H. Taavola and I. Taboada and A. Tamburro and A. Tepe and S. Ter-Antonyan and G. Tešić and S. Tilav and Toale, {P. A.} and Tobin, {M. N.} and S. Toscano and M. Tselengidou and E. Unger and M. Usner and S. Vallecorsa and {Van Eijndhoven}, N. and {Van Overloop}, A. and {Van Santen}, J. and M. Vehring and M. Voge and M. Vraeghe and C. Walck and T. Waldenmaier and M. Wallraff and Ch Weaver and M. Wellons and C. Wendt and S. Westerhoff and N. Whitehorn and K. Wiebe and Wiebusch, {C. H.} and Williams, {D. R.} and H. Wissing and M. Wolf and Wood, {T. R.} and K. Woschnagg and Xu, {D. L.} and Xu, {X. W.} and Yanez, {J. P.} and G. Yodh and S. Yoshida and P. Zarzhitsky and J. Ziemann and S. Zierke and M. Zoll",
year = "2015",
month = "1",
day = "1",
doi = "10.1016/j.jpdc.2014.08.001",
language = "English (US)",
volume = "75",
pages = "198--211",
journal = "Journal of Parallel and Distributed Computing",
issn = "0743-7315",
publisher = "Academic Press Inc.",

}

Aartsen, MG, Abbasi, R, Ackermann, M, Adams, J, Aguilar, JA, Ahlers, M, Altmann, D, Arguelles, C, Auffenberg, J, Bai, X, Baker, M, Barwick, SW, Baum, V, Bay, R, Beatty, JJ, Becker Tjus, J, Becker, KH, Benzvi, S, Berghaus, P, Berley, D, Bernardini, E, Bernhard, A, Besson, DZ, Binder, G, Bindig, D, Bissok, M, Blaufuss, E, Blumenthal, J, Boersma, DJ, Bohm, C, Bose, D, Böser, S, Botner, O, Brayeur, L, Bretz, HP, Brown, AM, Bruijn, R, Casey, J, Casier, M, Chirkin, D, Christov, A, Christy, B, Clark, K, Classen, L, Clevermann, F, Coenders, S, Cohen, S, Cowen, D, Cruz Silva, AH, Danninger, M, Daughhetee, J, Davis, JC, Day, M, De Clercq, C, De Ridder, S, Desiati, P, De Vries, KD, De With, M, Deyoung, T, Díaz-Vélez, JC, Dunkman, M, Eagan, R, Eberhardt, B, Eichmann, B, Eisch, J, Euler, S, Evenson, PA, Fadiran, O, Fazely, AR, Fedynitch, A, Feintzeig, J, Feusels, T, Filimonov, K, Finley, C, Fischer-Wasels, T, Flis, S, Franckowiak, A, Frantzen, K, Fuchs, T, Gaisser, TK, Gallagher, J, Gerhardt, L, Gladstone, L, Glüsenkamp, T, Goldschmidt, A, Golup, G, Gonzalez, JG, Goodman, JA, Góra, D, Grandmont, DT, Grant, D, Gretskov, P, Groh, JC, Groß, A, Ha, C, Haj Ismail, A, Hallen, P, Hallgren, A, Halzen, F, Hanson, K, Hebecker, D, Heereman, D, Heinen, D, Helbing, K, Hellauer, R, Hickford, S, Hill, GC, Hoffman, KD, Hoffmann, R, Homeier, A, Hoshina, K, Huang, F, Huelsnitz, W, Hulth, PO, Hultqvist, K, Hussain, S, Ishihara, A, Jacobi, E, Jacobsen, J, Jagielski, K, Japaridze, GS, Jero, K, Jlelati, O, Kaminsky, B, Kappes, A, Karg, T, Karle, A, Kauer, M, Kelley, JL, Kiryluk, J, Kläs, J, Klein, SR, Köhne, JH, Kohnen, G, Kolanoski, H, Köpke, L, Kopper, C, Kopper, S, Koskinen, DJ, Kowalski, M, Krasberg, M, Kriesten, A, Krings, K, Kroll, G, Kunnen, J, Kurahashi, N, Kuwabara, T, Labare, M, Landsman, H, Larson, MJ, Lesiak-Bzdak, M, Leuermann, M, Leute, J, Lünemann, J, Macías, O, Madsen, J, Maggi, G, Maruyama, R, Mase, K, Matis, HS, McNally, F, Meagher, K, Merck, M, Merino, G, Meures, T, Miarecki, S, Middell, E, Milke, N, Miller, J, Mohrmann, L, Montaruli, T, Morse, R, Nahnhauer, R, Naumann, U, Niederhausen, H, Nowicki, SC, Nygren, DR, Obertacke, A, Odrowski, S, Olivas, A, Omairat, A, O'Murchadha, A, Paul, L, Pepper, JA, Pérez De Los Heros, C, Pfendner, C, Pieloth, D, Pinat, E, Posselt, J, Price, PB, Przybylski, GT, Quinnan, M, Rädel, L, Rae, I, Rameez, M, Rawlins, K, Redl, P, Reimann, R, Resconi, E, Rhode, W, Ribordy, M, Richman, M, Riedel, B, Rodrigues, JP, Rott, C, Ruhe, T, Ruzybayev, B, Ryckbosch, D, Saba, SM, Sander, HG, Santander, M, Sarkar, S, Schatto, K, Scheriau, F, Schmidt, T, Schmitz, M, Schoenen, S, Schöneberg, S, Schönwald, A, Schukraft, A, Schulte, L, Schultz, D, Schulz, O, Seckel, D, Sestayo, Y, Seunarine, S, Shanidze, R, Sheremata, C, Smith, MWE, Soldin, D, Spiczak, GM, Spiering, C, Stamatikos, M, Stanev, T, Stanisha, NA, Stasik, A, Stezelberger, T, Stokstad, RG, Stößl, A, Strahler, EA, Ström, R, Strotjohann, NL, Sullivan, GW, Taavola, H, Taboada, I, Tamburro, A, Tepe, A, Ter-Antonyan, S, Tešić, G, Tilav, S, Toale, PA, Tobin, MN, Toscano, S, Tselengidou, M, Unger, E, Usner, M, Vallecorsa, S, Van Eijndhoven, N, Van Overloop, A, Van Santen, J, Vehring, M, Voge, M, Vraeghe, M, Walck, C, Waldenmaier, T, Wallraff, M, Weaver, C, Wellons, M, Wendt, C, Westerhoff, S, Whitehorn, N, Wiebe, K, Wiebusch, CH, Williams, DR, Wissing, H, Wolf, M, Wood, TR, Woschnagg, K, Xu, DL, Xu, XW, Yanez, JP, Yodh, G, Yoshida, S, Zarzhitsky, P, Ziemann, J, Zierke, S & Zoll, M 2015, 'The IceProd framework: Distributed data processing for the IceCube neutrino observatory', Journal of Parallel and Distributed Computing, vol. 75, pp. 198-211. https://doi.org/10.1016/j.jpdc.2014.08.001

The IceProd framework : Distributed data processing for the IceCube neutrino observatory. / Aartsen, M. G.; Abbasi, R.; Ackermann, M.; Adams, J.; Aguilar, J. A.; Ahlers, M.; Altmann, D.; Arguelles, C.; Auffenberg, J.; Bai, X.; Baker, M.; Barwick, S. W.; Baum, V.; Bay, R.; Beatty, J. J.; Becker Tjus, J.; Becker, K. H.; Benzvi, S.; Berghaus, P.; Berley, D.; Bernardini, E.; Bernhard, A.; Besson, D. Z.; Binder, G.; Bindig, D.; Bissok, M.; Blaufuss, E.; Blumenthal, J.; Boersma, D. J.; Bohm, C.; Bose, D.; Böser, S.; Botner, O.; Brayeur, L.; Bretz, H. P.; Brown, A. M.; Bruijn, R.; Casey, J.; Casier, M.; Chirkin, D.; Christov, A.; Christy, B.; Clark, K.; Classen, L.; Clevermann, F.; Coenders, S.; Cohen, S.; Cowen, Douglas; Cruz Silva, A. H.; Danninger, M.; Daughhetee, J.; Davis, J. C.; Day, M.; De Clercq, C.; De Ridder, S.; Desiati, P.; De Vries, K. D.; De With, M.; Deyoung, T.; Díaz-Vélez, J. C.; Dunkman, M.; Eagan, R.; Eberhardt, B.; Eichmann, B.; Eisch, J.; Euler, S.; Evenson, P. A.; Fadiran, O.; Fazely, A. R.; Fedynitch, A.; Feintzeig, J.; Feusels, T.; Filimonov, K.; Finley, C.; Fischer-Wasels, T.; Flis, S.; Franckowiak, A.; Frantzen, K.; Fuchs, T.; Gaisser, T. K.; Gallagher, J.; Gerhardt, L.; Gladstone, L.; Glüsenkamp, T.; Goldschmidt, A.; Golup, G.; Gonzalez, J. G.; Goodman, J. A.; Góra, D.; Grandmont, D. T.; Grant, D.; Gretskov, P.; Groh, J. C.; Groß, A.; Ha, C.; Haj Ismail, A.; Hallen, P.; Hallgren, A.; Halzen, F.; Hanson, K.; Hebecker, D.; Heereman, D.; Heinen, D.; Helbing, K.; Hellauer, R.; Hickford, S.; Hill, G. C.; Hoffman, K. D.; Hoffmann, R.; Homeier, A.; Hoshina, K.; Huang, F.; Huelsnitz, W.; Hulth, P. O.; Hultqvist, K.; Hussain, S.; Ishihara, A.; Jacobi, E.; Jacobsen, J.; Jagielski, K.; Japaridze, G. S.; Jero, K.; Jlelati, O.; Kaminsky, B.; Kappes, A.; Karg, T.; Karle, A.; Kauer, M.; Kelley, J. L.; Kiryluk, J.; Kläs, J.; Klein, S. R.; Köhne, J. H.; Kohnen, G.; Kolanoski, H.; Köpke, L.; Kopper, C.; Kopper, S.; Koskinen, D. J.; Kowalski, M.; Krasberg, M.; Kriesten, A.; Krings, K.; Kroll, G.; Kunnen, J.; Kurahashi, N.; Kuwabara, T.; Labare, M.; Landsman, H.; Larson, M. J.; Lesiak-Bzdak, M.; Leuermann, M.; Leute, J.; Lünemann, J.; Macías, O.; Madsen, J.; Maggi, G.; Maruyama, R.; Mase, K.; Matis, H. S.; McNally, F.; Meagher, K.; Merck, M.; Merino, G.; Meures, T.; Miarecki, S.; Middell, E.; Milke, N.; Miller, J.; Mohrmann, L.; Montaruli, T.; Morse, R.; Nahnhauer, R.; Naumann, U.; Niederhausen, H.; Nowicki, S. C.; Nygren, D. R.; Obertacke, A.; Odrowski, S.; Olivas, A.; Omairat, A.; O'Murchadha, A.; Paul, L.; Pepper, J. A.; Pérez De Los Heros, C.; Pfendner, C.; Pieloth, D.; Pinat, E.; Posselt, J.; Price, P. B.; Przybylski, G. T.; Quinnan, M.; Rädel, L.; Rae, I.; Rameez, M.; Rawlins, K.; Redl, P.; Reimann, R.; Resconi, E.; Rhode, W.; Ribordy, M.; Richman, M.; Riedel, B.; Rodrigues, J. P.; Rott, C.; Ruhe, T.; Ruzybayev, B.; Ryckbosch, D.; Saba, S. M.; Sander, H. G.; Santander, M.; Sarkar, S.; Schatto, K.; Scheriau, F.; Schmidt, T.; Schmitz, M.; Schoenen, S.; Schöneberg, S.; Schönwald, A.; Schukraft, A.; Schulte, L.; Schultz, D.; Schulz, O.; Seckel, D.; Sestayo, Y.; Seunarine, S.; Shanidze, R.; Sheremata, C.; Smith, M. W.E.; Soldin, D.; Spiczak, G. M.; Spiering, C.; Stamatikos, M.; Stanev, T.; Stanisha, N. A.; Stasik, A.; Stezelberger, T.; Stokstad, R. G.; Stößl, A.; Strahler, E. A.; Ström, R.; Strotjohann, N. L.; Sullivan, G. W.; Taavola, H.; Taboada, I.; Tamburro, A.; Tepe, A.; Ter-Antonyan, S.; Tešić, G.; Tilav, S.; Toale, P. A.; Tobin, M. N.; Toscano, S.; Tselengidou, M.; Unger, E.; Usner, M.; Vallecorsa, S.; Van Eijndhoven, N.; Van Overloop, A.; Van Santen, J.; Vehring, M.; Voge, M.; Vraeghe, M.; Walck, C.; Waldenmaier, T.; Wallraff, M.; Weaver, Ch; Wellons, M.; Wendt, C.; Westerhoff, S.; Whitehorn, N.; Wiebe, K.; Wiebusch, C. H.; Williams, D. R.; Wissing, H.; Wolf, M.; Wood, T. R.; Woschnagg, K.; Xu, D. L.; Xu, X. W.; Yanez, J. P.; Yodh, G.; Yoshida, S.; Zarzhitsky, P.; Ziemann, J.; Zierke, S.; Zoll, M.

In: Journal of Parallel and Distributed Computing, Vol. 75, 01.01.2015, p. 198-211.

Research output: Contribution to journalArticle

TY - JOUR

T1 - The IceProd framework

T2 - Distributed data processing for the IceCube neutrino observatory

AU - Aartsen, M. G.

AU - Abbasi, R.

AU - Ackermann, M.

AU - Adams, J.

AU - Aguilar, J. A.

AU - Ahlers, M.

AU - Altmann, D.

AU - Arguelles, C.

AU - Auffenberg, J.

AU - Bai, X.

AU - Baker, M.

AU - Barwick, S. W.

AU - Baum, V.

AU - Bay, R.

AU - Beatty, J. J.

AU - Becker Tjus, J.

AU - Becker, K. H.

AU - Benzvi, S.

AU - Berghaus, P.

AU - Berley, D.

AU - Bernardini, E.

AU - Bernhard, A.

AU - Besson, D. Z.

AU - Binder, G.

AU - Bindig, D.

AU - Bissok, M.

AU - Blaufuss, E.

AU - Blumenthal, J.

AU - Boersma, D. J.

AU - Bohm, C.

AU - Bose, D.

AU - Böser, S.

AU - Botner, O.

AU - Brayeur, L.

AU - Bretz, H. P.

AU - Brown, A. M.

AU - Bruijn, R.

AU - Casey, J.

AU - Casier, M.

AU - Chirkin, D.

AU - Christov, A.

AU - Christy, B.

AU - Clark, K.

AU - Classen, L.

AU - Clevermann, F.

AU - Coenders, S.

AU - Cohen, S.

AU - Cowen, Douglas

AU - Cruz Silva, A. H.

AU - Danninger, M.

AU - Daughhetee, J.

AU - Davis, J. C.

AU - Day, M.

AU - De Clercq, C.

AU - De Ridder, S.

AU - Desiati, P.

AU - De Vries, K. D.

AU - De With, M.

AU - Deyoung, T.

AU - Díaz-Vélez, J. C.

AU - Dunkman, M.

AU - Eagan, R.

AU - Eberhardt, B.

AU - Eichmann, B.

AU - Eisch, J.

AU - Euler, S.

AU - Evenson, P. A.

AU - Fadiran, O.

AU - Fazely, A. R.

AU - Fedynitch, A.

AU - Feintzeig, J.

AU - Feusels, T.

AU - Filimonov, K.

AU - Finley, C.

AU - Fischer-Wasels, T.

AU - Flis, S.

AU - Franckowiak, A.

AU - Frantzen, K.

AU - Fuchs, T.

AU - Gaisser, T. K.

AU - Gallagher, J.

AU - Gerhardt, L.

AU - Gladstone, L.

AU - Glüsenkamp, T.

AU - Goldschmidt, A.

AU - Golup, G.

AU - Gonzalez, J. G.

AU - Goodman, J. A.

AU - Góra, D.

AU - Grandmont, D. T.

AU - Grant, D.

AU - Gretskov, P.

AU - Groh, J. C.

AU - Groß, A.

AU - Ha, C.

AU - Haj Ismail, A.

AU - Hallen, P.

AU - Hallgren, A.

AU - Halzen, F.

AU - Hanson, K.

AU - Hebecker, D.

AU - Heereman, D.

AU - Heinen, D.

AU - Helbing, K.

AU - Hellauer, R.

AU - Hickford, S.

AU - Hill, G. C.

AU - Hoffman, K. D.

AU - Hoffmann, R.

AU - Homeier, A.

AU - Hoshina, K.

AU - Huang, F.

AU - Huelsnitz, W.

AU - Hulth, P. O.

AU - Hultqvist, K.

AU - Hussain, S.

AU - Ishihara, A.

AU - Jacobi, E.

AU - Jacobsen, J.

AU - Jagielski, K.

AU - Japaridze, G. S.

AU - Jero, K.

AU - Jlelati, O.

AU - Kaminsky, B.

AU - Kappes, A.

AU - Karg, T.

AU - Karle, A.

AU - Kauer, M.

AU - Kelley, J. L.

AU - Kiryluk, J.

AU - Kläs, J.

AU - Klein, S. R.

AU - Köhne, J. H.

AU - Kohnen, G.

AU - Kolanoski, H.

AU - Köpke, L.

AU - Kopper, C.

AU - Kopper, S.

AU - Koskinen, D. J.

AU - Kowalski, M.

AU - Krasberg, M.

AU - Kriesten, A.

AU - Krings, K.

AU - Kroll, G.

AU - Kunnen, J.

AU - Kurahashi, N.

AU - Kuwabara, T.

AU - Labare, M.

AU - Landsman, H.

AU - Larson, M. J.

AU - Lesiak-Bzdak, M.

AU - Leuermann, M.

AU - Leute, J.

AU - Lünemann, J.

AU - Macías, O.

AU - Madsen, J.

AU - Maggi, G.

AU - Maruyama, R.

AU - Mase, K.

AU - Matis, H. S.

AU - McNally, F.

AU - Meagher, K.

AU - Merck, M.

AU - Merino, G.

AU - Meures, T.

AU - Miarecki, S.

AU - Middell, E.

AU - Milke, N.

AU - Miller, J.

AU - Mohrmann, L.

AU - Montaruli, T.

AU - Morse, R.

AU - Nahnhauer, R.

AU - Naumann, U.

AU - Niederhausen, H.

AU - Nowicki, S. C.

AU - Nygren, D. R.

AU - Obertacke, A.

AU - Odrowski, S.

AU - Olivas, A.

AU - Omairat, A.

AU - O'Murchadha, A.

AU - Paul, L.

AU - Pepper, J. A.

AU - Pérez De Los Heros, C.

AU - Pfendner, C.

AU - Pieloth, D.

AU - Pinat, E.

AU - Posselt, J.

AU - Price, P. B.

AU - Przybylski, G. T.

AU - Quinnan, M.

AU - Rädel, L.

AU - Rae, I.

AU - Rameez, M.

AU - Rawlins, K.

AU - Redl, P.

AU - Reimann, R.

AU - Resconi, E.

AU - Rhode, W.

AU - Ribordy, M.

AU - Richman, M.

AU - Riedel, B.

AU - Rodrigues, J. P.

AU - Rott, C.

AU - Ruhe, T.

AU - Ruzybayev, B.

AU - Ryckbosch, D.

AU - Saba, S. M.

AU - Sander, H. G.

AU - Santander, M.

AU - Sarkar, S.

AU - Schatto, K.

AU - Scheriau, F.

AU - Schmidt, T.

AU - Schmitz, M.

AU - Schoenen, S.

AU - Schöneberg, S.

AU - Schönwald, A.

AU - Schukraft, A.

AU - Schulte, L.

AU - Schultz, D.

AU - Schulz, O.

AU - Seckel, D.

AU - Sestayo, Y.

AU - Seunarine, S.

AU - Shanidze, R.

AU - Sheremata, C.

AU - Smith, M. W.E.

AU - Soldin, D.

AU - Spiczak, G. M.

AU - Spiering, C.

AU - Stamatikos, M.

AU - Stanev, T.

AU - Stanisha, N. A.

AU - Stasik, A.

AU - Stezelberger, T.

AU - Stokstad, R. G.

AU - Stößl, A.

AU - Strahler, E. A.

AU - Ström, R.

AU - Strotjohann, N. L.

AU - Sullivan, G. W.

AU - Taavola, H.

AU - Taboada, I.

AU - Tamburro, A.

AU - Tepe, A.

AU - Ter-Antonyan, S.

AU - Tešić, G.

AU - Tilav, S.

AU - Toale, P. A.

AU - Tobin, M. N.

AU - Toscano, S.

AU - Tselengidou, M.

AU - Unger, E.

AU - Usner, M.

AU - Vallecorsa, S.

AU - Van Eijndhoven, N.

AU - Van Overloop, A.

AU - Van Santen, J.

AU - Vehring, M.

AU - Voge, M.

AU - Vraeghe, M.

AU - Walck, C.

AU - Waldenmaier, T.

AU - Wallraff, M.

AU - Weaver, Ch

AU - Wellons, M.

AU - Wendt, C.

AU - Westerhoff, S.

AU - Whitehorn, N.

AU - Wiebe, K.

AU - Wiebusch, C. H.

AU - Williams, D. R.

AU - Wissing, H.

AU - Wolf, M.

AU - Wood, T. R.

AU - Woschnagg, K.

AU - Xu, D. L.

AU - Xu, X. W.

AU - Yanez, J. P.

AU - Yodh, G.

AU - Yoshida, S.

AU - Zarzhitsky, P.

AU - Ziemann, J.

AU - Zierke, S.

AU - Zoll, M.

PY - 2015/1/1

Y1 - 2015/1/1

N2 - IceCube is a one-gigaton instrument located at the geographic South Pole, designed to detect cosmic neutrinos, identify the particle nature of dark matter, and study high-energy neutrinos themselves. Simulation of the IceCube detector and processing of data require a significant amount of computational resources. This paper presents the first detailed description of IceProd, a lightweight distributed management system designed to meet these requirements. It is driven by a central database in order to manage mass production of simulations and analysis of data produced by the IceCube detector. IceProd runs as a separate layer on top of other middleware and can take advantage of a variety of computing resources, including grids and batch systems such as CREAM, HTCondor, and PBS. This is accomplished by a set of dedicated daemons that process job submission in a coordinated fashion through the use of middleware plugins that serve to abstract the details of job submission and job management from the framework.

AB - IceCube is a one-gigaton instrument located at the geographic South Pole, designed to detect cosmic neutrinos, identify the particle nature of dark matter, and study high-energy neutrinos themselves. Simulation of the IceCube detector and processing of data require a significant amount of computational resources. This paper presents the first detailed description of IceProd, a lightweight distributed management system designed to meet these requirements. It is driven by a central database in order to manage mass production of simulations and analysis of data produced by the IceCube detector. IceProd runs as a separate layer on top of other middleware and can take advantage of a variety of computing resources, including grids and batch systems such as CREAM, HTCondor, and PBS. This is accomplished by a set of dedicated daemons that process job submission in a coordinated fashion through the use of middleware plugins that serve to abstract the details of job submission and job management from the framework.

UR - http://www.scopus.com/inward/record.url?scp=84915776438&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84915776438&partnerID=8YFLogxK

U2 - 10.1016/j.jpdc.2014.08.001

DO - 10.1016/j.jpdc.2014.08.001

M3 - Article

VL - 75

SP - 198

EP - 211

JO - Journal of Parallel and Distributed Computing

JF - Journal of Parallel and Distributed Computing

SN - 0743-7315

ER -