Fun with keywords and data mining

One of the baffling qualities of digital media is its liquidity. Once something is encoded in binary code, we have endlessly interesting access to not just single creative works, but multiple works and even databases, along with the ability to search and see the parts and the whole in wonderful and new ways.

A new fascination for me in that regard is the ‘weighted word list,’ a visual representation of keywords (found in a document, in a book, in data sets, or on the entire Internet) that shows both the most common words and their relative frequency (more frequent words are larger, less frequent words are smaller). It’s a contemporary cousin of those old, dusty concordances (counting word frequency in the works of Shakespeare, for example).

Weighted word lists are popping up everywhere nowadays — from the new ‘tagging’ function of Technorati, to the web site interconnection analysis of Findory Neighbors. These visualizations provide a quick and intuitive view of complex data and the relationships between them.

So, why do I care? And why should you? The liquidity of information on-line has vast potential for helping us discover patterns, trends, connections, and disconnections we might not otherwise see. Imagine, for example, if Ticketmaster offered a weighted word list of ALL performers on sale nationwide, with the text size determined by total ticket sales? What an interesting constellation that would provide. Or, what if you could generate a weighted word list of the content from ALL of your marketing materials, and compare it to a similar list of ALL customer comments and feedback over the past year? You might just see some interesting patterns between what you sell, and what they connect to (or you might see some disconnects).

As an experiment, I’ve created a weighted word list of my own, compiling every word from this weblog’s history, and arranging the most common words in graphic format. It’s all done automatically and on the fly (the list rebuilds every time you load the page, adding every new weblog post as they come). If I’m doing my job, the BIGGER words should have some relation to my declared subject, and the patterns of words should reflect my common themes. Have a look and decide for yourself.

Thanks to dan & sherree for the base code for this function, which I modified a bit for use on this site.