US Department of Energy: Automated Sensor Data Processing with FileWatcher

The Challenge

The US Department of Energy operates a network of national laboratories where thousands of scientific instruments and environmental sensors generate continuous streams of data files. These files arrive at irregular intervals in various formats across dozens of monitored directories on Windows-based data acquisition servers. Previously, laboratory technicians manually checked directories and initiated processing scripts, leading to delays in data availability for researchers, missed files during off-hours, and inconsistent processing schedules. The department needed an automated monitoring solution that could run unattended as a Windows service, detect new files within seconds of arrival, trigger format-specific processing pipelines, and provide reliable operation across facilities without requiring dedicated operations staff for each data collection point.

The Solution

The Department of Energy deployed FileWatcher across data acquisition servers at multiple national laboratories. FileWatcher runs as a Windows service, continuously monitoring designated directories for new sensor data files. When files arrive, FileWatcher triggers customised action chains based on file type and source directory: CSV sensor readings are validated and forwarded to analysis databases, binary instrument dumps are queued for TextPipe Pro conversion, and image files from monitoring cameras are compressed and archived. FileWatcher's built-in FTP upload capability transfers processed results to centralised data repositories, while its service-mode operation ensures automatic restart after system updates or unexpected interruptions. The configurable retry logic handles temporary network outages gracefully, and detailed event logging provides complete audit trails for regulatory compliance across all facilities.

Results

Products Used

• FileWatcher — Continuous folder monitoring, automated file detection, action chain triggering, FTP upload, and Windows service operation
• TextPipe Pro — Binary instrument data conversion triggered by FileWatcher