The Department of Energy (DOE) Office of Science (SC) facilities
including accelerators, light sources and neutron sources and sensors
that study, the environment, and the atmosphere, are producing
streaming data that needs to be analyzed for next-generation
scientific discoveries. There has been an explosion of new research
and technologies for stream analytics arising from the academic and
private sectors. However, there has been no corresponding effort in
either documenting the critical research opportunities or building a
community that can create and foster productive collaborations. The
two-part workshop series, STREAM: Streaming Requirements, Experience,
Applications and Middleware Workshop (STREAM2015 and STREAM2016), were
conducted to bring the community together and identify gaps and future
efforts needed by both NSF and DOE.
This report describes the discussions, outcomes and conclusions from
STREAM2016: Streaming Requirements, Experience, Applications and
Middleware Workshop, the second of these workshops held on March
22-23, 2016 in Tysons, VA. STREAM2016 focused on the Department of
Energy (DOE) applications, computational and experimental facilities,
as well software systems. Thus, the role of “streaming and steering”
as a critical mode of connecting the experimental and computing
facilities was pervasive through the workshop. Given the overlap in
interests and challenges with industry, the workshop had significant
presence from several innovative companies and major contributors.
The requirements that drive the proposed research directions,
identified in this report, show an important opportunity for building
competitive research and development program around streaming
data. These findings and recommendations are consistent with vision
outlined in NRC Frontiers of Data and National Strategic Computing
Initiative (NCSI) [1, 2]. The discussions from the workshop are
captured as topic areas covered in this report's sections. The report
discusses four research directions driven by current and future
application requirements reflecting the areas identified as important
by STREAM2016. These include (i) Algorithms, (ii) Programming Models,
Languages and Runtime Systems (iii) Human-in-the-loop and Steering in
Scientific Workflow and (iv) Facilities.