Search and monitoring
Search extract data from data monitoring calls for an action from data.
Search is transient and leaves no trace. Monitoring is a constant execution of the same search (or multiple searches) and each instance changes the data.
When someone searches Google for a term, Google runs a query and returns a list of items than matches as close as possible the submitted term. If you run the same search again you may get the same or maybe slightly different results, it depends on how frequent Google refresh the data. Each search is independent of each other. Unless Google keeps record of our searches for statistics there is not real trace of a search.
In a way search is like the roulette game it has no memory.
Monitoring is more like the blackjack game. It has memory. In my world we refer to memory as state and monitoring as a state-full search. Maybe this is why I think that monitoring is way more interesting than a single search (btw, I don’t play either game but at least the later give you some chance)
Every executed search changes the state and the the starting point for the next search. So what is this state?
The state is the subject’s status and it could have three possible option of being:
- No state – we know nothing about it
- Intermediate state – we know something about this item but it is not enough to report about it
- Completion – there is something interesting to report about this subject
The subject toggles between these three options.
The subject moves from No State to Intermediate state when we find new data that worth keeping.
The subject moves from No State to Completion if we find in one iteration all the desired data.
The subject moves from Intermediate state to Completion if we find the rest of the desired data.
The subject moves from Intermediate state to No State if the stored data ages (and now obsolete) or we don’t care anymore.
The subject moves to No State as soon as it reaches Completion.
Actually, in some cases there is no need to store the complete state.
When running search query one expect all the results to appear at the same time. Monitoring works like precipitation in Chemistry results accumulate like little water drops over time. Like in Chemistry where it is used to separate solid from a solution we use it to extract meaningful data out of the “cloud”.
Monitoring is only one application of state-full search. Actually, it is the execution of this kind of search in the present. State-full search exist in each of the three tenses. It has different form and applications for each tense.
You can use state-full search in the past, e.g. stock trader, counting on technical analysts are using it for back testing, proving that their early indicators for a trend shift are correct.
Another use of state-full search in the present is personalization. Your purchase history could predict your future spending and Amazon offer products this way as soon as you enter their web-site.
Using state-full search in the future for simulation is another application. Playing “what if” scenarios.
The three main challenges in state-full search is to decide what data to keep, how to organize the data, and debugging. Saving too little and it is hard to find something meaningfully. Saving too much beats the purpose of filtering information. Organizing the data the “wrong” way will kill performance. Both problem are hard to fix in the “middle” of a search. The debugging challenge refers to the ability to explain why this is the right results or why there is no result. It is not always intuitive why certain state changed and like in most software programs you can be surprised by a new path of execution. Each of these challenges impact the key for a successful state-full search implementation: flexibility.
In my opinion the most significant difference between search and state-full search is the results.
Data + search = data
Data + state-full search = events, leads, alerts, actionable items, new data
Love to hear your thoughts!