Mass-market software able to do this is still nascent. Learning what that software is, what it will be able to do, is still being developed and understood. But by using the human learning analogy, we can start recognizing how intuitive pattern-based insight can come about using information technology. To start that, let's see where patterns come from in human perception.
The BasicsEven as early as the kindergarten level, children are taught to recognize patterns. Rhyme schemes in poetry, stripes of color on a piece of cloth, those are patterns a 5-year old can identify with the following steps:
Sensing/Acquisition
Identification/Features
The child takes contextual information and extracts relevant sections of data, or features. For example, hearing or reading the following poems, there is a vast amount of processing going on in the brain to convert dots on a page or sounds in the air into words.
Roses are red,
Violets are blue,
Sugar is sweet;
And so are you.
(from Solitude, by Alexander Pope)
Happy the man whose wish and care
A few paternal acres bound,
Content to breathe his native air
In his own ground.
Memory/Familiarity
The child has been introduced to the concepts before and remembers them. The pattern types have names or identifiers, which means they are special enough to identify with their own terminology. Once the words have been extracted by eyes and/or ears, training, instruction, and having seen the concepts before makes reducing the rhyme scheme to ABAB a “simple task”.The poems are drastically different (different words and themes), but the ABAB pattern is fundamental to poetry. If you are searching for the ABAB pattern in a poetry search engine, both of these poems would come back.
Visualization
Patterns are fundamental components of visualization. A Google search for pattern images (http://www.google.com/images?q=patterns ) yields images like these:
In some sense, many advanced patterns, especially those encoded in graphs and charts, are built on the fundamentals described in the previous steps. Seasonal sales trends, stock market/financial market data, radiology images, with this kind of data there are shapes or trends that the human in the loop can understand. Any software that can capitalize on the innate human ability to understand a pattern within data once it’s been recognized, and still use that exact same pattern to make decisions in an automated fashion as computers are very good at doing is key to making pattern based analytics a “must-have” tool in the analyst’s toolbox, regardless of what their data might actually contain.
A better definition of a pattern would be something that repeats somehow in a predictable way. The predictable part of that definition is very important and is what makes patterns so powerful. If you can correctly generalize what you are seeing then you can guess what’s coming next. That is also how you ensure that your pattern is correct. Recognizing a pattern is good but you can’t know it’s valid until you make a prediction with it and then test it.
ReplyDeleteFor example, if you place n points around the edge of a circle and draw all the lines connecting every pair of points how many regions will you get? Note that you can’t have more than two lines meet at any point in the circle. For 1 point there are no lines so 1 region, for 2 points the circle is divided in two so 2 regions, for 3 points you get 4 regions, for 4 points 8 regions and 5 points 16 regions. At that point you would start to see the pattern 1, 2, 4, 8, 16 and if I asked you what came next you might say 32, 64, 128 i.e. the powers of 2. This is the obvious and simplest model. But it turns out that the next number for 6 points is 31 and 7 points is 57. In this case, the initial pattern is wrong because it doesn’t predict the pattern properly because it is too simple. It turns out that this sequence is the sum of the first 5 binomial coefficients for power (n-1) i.e. m CHOOSE 0 + m CHOOSE 1 + m CHOOSE 2 + m CHOOSE 3 + m CHOOSE 4 where m is n - 1.
The point is that patterns can be powerful predictive tools but once you think you have a pattern you can’t be confident that it is the right pattern until it makes a prediction and proves that it is the right pattern for the situation.
Su, Francis E., et al. "Misleading Sequence." Math Fun Facts.
Perhaps the inverse of humans being so good at learning and using patterns is that they are not good at being random. When humans attempt to generate random data, the data produced is still being determined by their known behaviors. If other humans are paying close enough attention, which usually means there are financial incentives, they will eventually be able to determine the pattern. One of the more famous examples of this was with the game show Press Your Luck. A contestant was able to discern a pattern in the game play, which was thought to be randomized, and gain an advantage [http://en.wikipedia.org/wiki/Press_Your_Luck#Michael_Larson]. Even with modern computer systems available to assist humans with producing random behaviors, this continues to be a problem. Just last month, a pattern was found in scratch off lottery tickets [http://www.lotterypost.com/news/227079/1940460]. In each of these examples, it is important to note that the patterns were discerned by humans. Computer systems are still being trained in the foundation of patterns to assist humans with such arbitrary pattern recognition.
ReplyDeleteYes, patterns are intrinsic to human experience, and a primary way that we navigate through life. It is interesting that different people follow different patterns. What is a good pattern for one person is not necessarily a good pattern for someone else. In science, we hope that the patterns that we observe and codify are fundamental in that they can generalize to new observations. An interesting question is how patterns form in social sciences and business? Are there group effects that drive pattern generation in these cases? And are the resulting patterns fickle like so often in human behavior, or more robust as in physical systems? The tension between pattern generation and pattern sustainability lies at the heart of many disciplines, offering significant opportunities for the development of a comprehensive pattern based analytics framework that reflects the behavior of a variety of systems in the world around us.
ReplyDelete