This week we began working on the second phase of our project involving anomaly and interesting event detection. Anomaly detection aims to detect time periods when sensors are not working properly, whether it be missing data or incorrect readings. Generally, anomalies pertain to sensor maintenance by monitoring the health of the sensor network. Interesting events includes environmentally induced events such as increasing high tide levels due to rainfall events. These kinds of events will be flagged for further investigation into the underlying causes.
Our overall approach is to choose a baseline for ‘normal’ tide patterns for each sensor and to compare new readings to this baseline. We will then calculate the residuals between the baseline and the new readings to determine if there is an interesting event/anomaly occurring.
A challenge we encountered during anomaly detection/interesting event detection is finding a reliable baseline tide pattern. We considered using published astronomical tide predictions, but a lot of resources online prohibit use of their data other than on their site. We also plotted the predicted tide patterns through the NOAA Fort Pulaski gauge, and although this served as a fairly accurate ground truth for our own Fort Pulaski sensors, there would need to be some adjustment calculations if we were to use the NOAA gauge as a ground-truth for our other sensors to account for differences in inland distance and tide magnitude. The above graph shows the NOAA gauge predictions (in blue) with our own Pulaski sensor data (orange), and we can see that they both align relatively well. In considering the possibilities for a reliable baseline, we are tentatively moving towards fitting a sine function that is able to capture the tide patterns, the primary benefit of this method being that we would have a fitted sine function for each sensor. Any deviations from this fitted function would be classified on a range of possible “interesting” events.
This week we also gave a mid-program presentation to our fellow interns, Chris Le Dantec, and invited guests. We got feedback on our preliminary visualizations and approaches to anomaly/interesting event detection. Moving forward, there is a lot of work left to be done, but we have a clear vision for what we need to do to complete our project in the coming weeks.