Outta here

Everything from this blog (pdf) but I will leave it here for now, maybe forever !

Thanks for stopping by ! If you would like to talk about any of this, feel free to get in touch.

Financial Visualizations of text

Don’t know how many posts I’ve now done entitled financial visualizations but I forgot to include  this in the previous post.  At the beginning of this project, decided that it would be interesting to look at use of words, tone, style of news and explore how different visualizations can create different meanings. With Many Eyes and Tagcrowd (as referred to on previous posts) there are now quick ways of finding and displaying word frequencies from a large block of text. Many Eyes wordtree also provides an alternative way by showing the context of the sentence surrounding a word e.g using the word fear gives an instant clue about possible context but using the full news sample shows a variety of possible meanings:

 

Many Eyes Wordtree

Many Eyes Wordtree

There could be a completely separate project just to look at individual words within wordtrees and provide analysis.
The two Surrey student projects mentioned previously studied use of text in two papers – BiModal Visualisation – A financial trading case study explored the use of sentiment as part of  the FINgrid / SATISfi projects. Summarization of Multimodal Information  explored ways of summarizing text and numeric series of data. They outline situations where for example, traders may be looking at numeric data and noting significant points then using financial news summaries, as opposed to simply headlines to check on before and after these points. They looked at six financial news stories daily during May 2002 and compared to daily market data.   The SATISfi visualization showed a one sentence summary plotted against a time series of market data including market data swings*.

Catherine Mulbrandon (2004) submitted her fascinating  and wonderfully clear thesis Visualizing Economics: Designing a Persuasive Argument exploring the design and presentation of economic data, either in or out of context and how graphics can distract instead of reveal the possible significance of economic data. She has continued her research on her blog Visualizing Economics. I especially like the idea that she focuses on items which are not in economic / financial specific press (this does seem to be where a lot of research such as the above two papers are completed) but the wider press where public policy is debated, great for anyone interested in the global public reaction to the media and how this may influence decision-making. 

She looks at the role of integrity in design and not just placement of text and graphics but use of size, colour etc can affect how an audience will judge the data.  She provides a number of examples where what is visualized can be distorted from what the data is actually about.

I think there are similarities in the use and placement of words in headlines, news articles too. Looking at word frequencies will provide some indication of a possible movement (trend doesn’t quite seem to be the right word) and use of descriptive words – are these just for entertainment or is there a conscious / subconscious attempt in the writing / editing of a media item to influence the reader ?

For example in the data I looked at

 

March – highest frequencies include – fall/fallen (7), recession (3), shock (2), decline (2), crisis (2), fear (2), higher (2), better (2) plunge (2) reassure (2).

Others with only (1) but more descriptive include – anxiously, chaos, dropped, downgraded, engulfed, jitters, knocked, lower / lowest, nail-biting, predicted, perversely, plummeted, raging, rocketing, rushed, rumours, seesawed, sharp, speculation, spiral, struggling, tumbled, turmoil, unravelling, wiping, worried, worst

Most of the above descriptive words although only featured once in a range of articles could be seen as mostly negative or indicating uncertainty, tension.  For some this might provide an opportunity to influence market trading and behaviour.  These are not boring words that would leave you without any trace of emotion (unless you were regularly exposed to all of them I guess).

Wordle is another interesting one – Many Eyes also incorporated it into its data visualizations last year. One reason I am interested in Wordle is because the creator of the visualization can choose how horizontally or vertically to present the words. So could a reader rightly or wrongly pick up something because a word is presented vertically and written upwards or downwards (combining the functionality of a line visualization as well as a word)?

e.g.

 

Lehman wordle

Lehman wordle

Think that’s it for now. Each one of these could be analysed in greater detail as separate visualization projects – I am not doing them justice at all. Quite frustrating now – would like to write more – e.g. on the visualization post earlier today, would happily do separate sections on just – shape, and colour – but not sure what to do and what use it will be – totally selfish but as with writing about finance, reaching a point now where don’t know what to put, need to be asked questions in order to focus this in a better way.

 

*It wasn’t until I met Lee in January that I found out he had done stuff with financial visualization, I knew he had something to with grid computing but my brain had failed to make the obvious (as he pointed out to me) connection that what kinds of systems would require grid – sized amounts of computing power – but hey, better late than never :-)

Financial Visualizations – sorry, long post & there are quite a few links

Trying to pull this together now – some of these don’t have permission to reproduce so linked.  Haven’t had time yet to really think through attention and memory aspects yet of this project as per mind map but that doesn’t need to be Many Eyes specific. Just wanted to use this post as a reference.

Still trying to think what works well (for both display on a monitor and web browser,  also page loading time, as well as how the brain may process it) or is a variety of these better (multiple devices, multiple brains) – diagrams with pastel colours, diagrams with bolder / sharper contrasting colours, use of photographic images in diagrams,  use of 3D using both, use of animation, use of video etc etc

Also with financial visualization, as per the finance page just created – what needs to be visualized.  Based on everything I have looked at in the last 2 /3 months, there are huge amounts of data visualization in the financial world. Overall stocks, shares, rates, prices are visualized, movements are visualized – but – the majority of things that are traded in the markets are not visible objects – contracts, paper, bonds etc etc I wonder if that makes a human less emotionally attached to the kinds of things that are being traded – because they are not visible, how much of a connection is there – and if the majority of a trading role is looking at volumes and movements…. I don’t know if there is any way to bring – emotion, more human-ness into financial visualization and whether it would produce a positive benefit?

Some examples of  visualizations I recently created:

Many Eyes visualizations

Many Eyes visualizations

 Also, thinking about info ‘clutter’ how eyes process information they are viewing, what is the difference between seeing multiple visualizations and viewing one at a time e.g.

Many Eyes single visualization

Many Eyes single visualization

Many Eyes has the option of a matrix chart which my own lack of data visualization ability means that I have not been able to use it effectively yet. I found one example of a matrix type chart in the WEF Financial Architecture full report Page 49, figure 27 which was useful. I like all the charts in this document actually.

IMF attempt to provide variety with map or bubble chart options

Some other possible financial visualizations or visualizations used in the financial world,  which I haven’t really had much chance to look at but are used:

Heatmaps – The Times , Paul Kedrosky – Infectious Greed one  ,  Financial Times  - shares etc , Heatmap explorer (not downloaded  / looked at this yet)

Candle charts – Lee (another Surrey lecturer) had students complete projects in around 2003 / 4 I think, looking at origins of financial data visualization (FINgrid is one of his personal projects – massive data & news collection, aggregation and visualization but I don’t know what it looks like other than screenshots on Surrey website), including Japanese rice trading – where they used to use actual rice grains to show opening, closing prices etc

I’m sure there are other types I’ve forgotten to mention. Forex have tools (not free)  The pattern site (not free) is more financially specific, but there is also a pattern browser project.  This chart chooser also provides useful information about what types of ones work well.

Use of arrows can be helpful. Why this in a way doesn’t feel like an attractive visualization (also the green / red – visibility contrast for so-called colour blindness or whatever it is called now)  although it is produced by WallStats  because as a PowerPoint using generation we got rid of clip-art, no clip-art  or boring use of flat colour in diagrams in our presentations anymore…just photographic art… its more professional….. It combines both, some part of me wants to scream from a personal design point of view, it shouldn’t work – but for me it does work – very well. Certainly the use of humour, you can identify with personalities – some simple use of text and arrows. Its a visualization which tells a story but not really all the complex data that has been involved ? I’m not sure if that sentence makes sense.

Here are some chart visuals with photos 

http://www.fotosearch.com/photos-images/bar-chart.html

Just looking at the page, I think (can’t say for sure) that my eyes moved towards the more – glowing, photographic ones.

The BBC combines in audio with photographs as a slideshow I think multimodal presentation may help with sense-making, there is only so much information that can be presented either on a computer screen, a mobile device, or even the huge trading floor screens – and there is only so much visual information that a brain can process in one go. I don’t know enough about how combining audio with a rich visual experience can enhance understanding of complex data – but I’m sure it must do. (Will refer back to Mind Hacks – which has loads of info about this, maybe in a future post).

Some other ways of visualizing financial data:

http://flowingdata.com/2009/01/01/9-ways-to-visualize-consumer-spending/

This design project for the Morning Star attempted to tackle financial visualization complexity – it is visually amazing to look at – more sophisticated than many others I have seen. I showed to Osama (surrey lecturer) whilst we were talking through all the visualizations and he said he liked it but it was too complex for him – he looked at the different images but couldn’t instantly grasp what they were trying to show. In a way I feel the same. I read a paper  last year on how to designing and density but I don’t think I really get it just yet.

I am finding this with things like Wikiviz and Skyrails too – I LOVE looking at them – in fact completely visually wowed by them, but I also find them very complex. They are able to take huge amounts of complex, interlinking data (global markets and other financial data would sit in that category too) and present it in a way that gives something of an instant visual clue but requires more digging in deeper to make more sense of it? So what is the ideal of the WallStats ones, ManyEyes, huge data cluster visualizations – which ones work well and where ?

2 new pages and feedback from Surrey (UPDATED)

Whilst I’m still results gathering and discussing, starting to put ideas on these two pages – linked from top of blog – visualization and finance

Also on second note – following chat today,  he absolutely loved them – regular & wikified !

He believes there is a lot of possibilities for using regular Many Eyes as well in similar kind of way – to explain efficient market hypothesis. He hadn’t seen Many Eyes before – so going to play with a bit. He can also see lots of ways for other things, we’re going to hopefully experiment before a mortgage lecture on 13th Feb. He is also keen on the wikified version because of the collaboration options as opposed to the ones available with regular Many Eyes. Very excited too about possibilities of using with live data – he hadn’t seen Google spreadsheets before today, so we’re going to try with that and see if we can get live updates – Yay !

Results – attempt 1 (long post)

Apologies but no pictures yet, its a long post, available as pdf  

A kind of rationale for doing this

Erm – well it seemed like a good idea in November.  Although this has gone way beyond a regular ‘software test’ – i.e. Many Eyes Wikified – should have been in and out within 24 hours but I’m glad to still be able to get in.

 The negative – at the time of writing, I’m not sure whether it has been an expensive and time-consuming ‘project’ which has not proved anything of much value and produced very little to move forward with (well duh, it wouldn’t – I wasn’t very hopeful, it was going to be impossible to do this much justice as a mostly individual effort but having begged my way onto the Many Eyes beta ..) However when I see Osama tomorrow at Surrey to talk about it, maybe will feel differently. Also whilst it’s useful in some ways to go into this completely open ended – probably doing something more in terms of having a plan and then sticking to it would have been good – maybe.

Personally its been painful, and sometime I would like to be able to sleep again instead of thinking about the state of the world every night (she says with great dramatic overtones) but I’m also glad that I have taken the time to try and understand / update more about the complexities of economics, economic theory etc Well that’s what I keep telling myself anyway.

 The positive – somehow seemed to have steered my way albeit without navigation (other than a couple of wonderful people who have offered advice and very useful sources of information) and bumped into the right kinds of things to help develop understanding of what 21st century economics is all about (so some of A level economics must have sunk in 1000 yrs ago when I studied it).  My SME has provided some useful insights but have been left pretty much to get on with it, having given the impression that I kind of knew enough to get going with it, he just asks for updates ;-)

That said – idea was originally to try and see if there was anything useful could do with visualizing historical news and market activity on a wiki – firstly to see how it could be presented, ease of use – data collection, aggregation and visualization; and secondly if there were any meaningful relationships identified could a wiki and visualizations on a wiki be used to create and analyse possible market scenarios or other global economic trends rather than just market-specific data only. SME has asked about possibilities of extending it further as a possible ‘tool’ e.g. bringing in RSS and live data – but haven’t been able to do yet due to lack of time.

I originally suggested this idea to said SME, bit of a guess but had decided already to choose Lehman Brothers because I felt it would give a good idea of the overall financial picture – as well as their own activity, their relationships to all areas and echelons of the financial world – this has turned out more enlightening than I had expected. I wasn’t paying sufficient attention to the press last September to realise that they were quite unique in terms of their levels of risk and that they were not bailed out.

Rationale for dates chosen:

  • March – Bear Stearns, Marubeni fraud, speculation about Lehman being next to go due to their subprime exposure (massive write-offs which would need to be balanced against – something), short selling accusations and Lehman recovery on 1st April.
  • June – some speculation around potential Lehman bidders, info about their assets
  • Sept – the dates prior to decision weekend when decision makers chose not to bail them out.

Other possible dates could have been mid-July when Royal Bank of Canada appeared to be a potential bidder but withdrew and also in August / early September when discussions opened with Korea Development Bank which fell through (although this is partly covered above).

Keywords:

Wanted to look at possible relationships of word use, frequency – ideally it would be reviewed against time of market data and post publication, but is limited as per the above. I used Many Eyes and Tagcrowd for this to get a range of visualization options.

Visualizations:

Wanted to create a wikified sub-dashboard (if using regular Many Eyes, imagine topic page with 6 boxes with snapshot of visualization in each one) trying to replicate possible views that someone with a daily interest in the financial world could see – also set up a Pageflakes page (as well as it being a reference source) due to Pageflakes showing a ‘dashboard’ type format with a mix of photographs, diagrams and headline text. A typical Many Eyes wikified page could display snapshots of at least 10 Java based visualizations – download time less than 1 min on broadband or faster. Post to follow in more detail about visualizations including review of shots of typical trading screens etc

As per mind map, if time will do a separate ‘presentation’ of thoughts about finance and economics.

General market data:

The problems with obtaining and visualizing the data – as documented previously – Financial Times, for overall tables – is useful but unable to export out easily with my level of ability / understanding. It is possible to export most of this data from Yahoo Finance  into spreadsheets but to get overall tables as in the FT data, it would require lots of re-ordering – I don’t know if there is a way of doing that automatically or writing something clever that can get it out of Yahoo Finance in this kind of format.

There were market swings (however I haven’t looked at market data for the last 10 yrs so have no idea whether what I’m calling a swing is much of one or not – hoping that SME will shortly be able to give an idea of this). For some dates there is incomplete information available during chosen periods. Highest swings were in gold and oil during period – oil rose sharply then fell below rates at beginning of 2008, gold rose & fell but overall declined, Gradual decline of most US & UK interest rates, others mixed. Some equity indices slight swings but overall equities declined. Most currencies declined overall.

News:

Economist, The Times (UK), Financial Times, The Telegraph (UK).

Areas not covered which would be interesting to look at – Factiva, blogs & comments, comparing these samples to complex data analysis e.g. Reuters  algorithms and FINgrid data for the same period (no access to), more detailed regulatory timescales. Also market data including Lehman shares against time of articles (current datasets are not sufficient for this).  Entries in terms of popularity – common examples of this in the financial press use heatmaps e.g., 1 , 2 , 3 to show popular articles / posts – don’t know if feasible for Many Eyes to have heatmap – as a variant of the tree-map chart or something else.

A lot of the news during the selected dates is mostly reporting of events without commentary, the articles contain some predictions from e.g. Lehman Brothers’ staff such as Fed rate cuts; they also expected shares to bottom out after eight months and predicted a market recovery in November. They expected oil to continue to rise, housing market to continue to decline. They encouraged investment in municipal bonds due to yields in April and suggested that credit concerns were due to misunderstandings. Lehman announced plans to set up a dark pool investment partnership (dark pool = less regulated, private platforms that trade shares away from regular exchanges). They also appear to have affected value of Freddie Mac and Fanny Mae by providing a report to show what was needed to save them which in turn crashed their own share value further.

Not much reported about financial regulation but ones during March – Sept 08 include legislation voted on so that financial advisers – mortgage lenders / other banks are required to show more detailed information on their balance sheets that could previously be used in financial transactions without being shown, this resulted in speculation that the financial crisis could worsen because investors not being provided with full information about their portfolios so more fearful of investing. Short selling (The selling of a security that the seller does not own, or any sale that is completed by the delivery of a security borrowed by the seller ) was also temporarily banned in the US and UK. Not including TARP (debt bailout) legislation due to being outside sample period.

March

Market data:

  • 11/03 most equities indices, currencies and interest rates up, oil up, gold down. As getting nearer 31/03 indices mixed, gold up, oil down, interest rates down.
  • Lehman shares – jump over weekend 14/03 – 17/03 with open 14/03 45.9, close 39.3, and 17/03 25.4 then 31.8. But 18/03 open 38, closing 46.5; then rising to 20/03 42.8, 48.6 respectively. 31/03 37.3, 37.6 and 01/04 40.9, 44.3

News:

  • 16/03 A handful of Wall Street investors are being blamed for driving up the cost of crude to record highs. Overall, hedge funds now account for almost 10% of the world’s oil trade, according to figures from the US Commodity Futures Trading Commission. “The numbers that came out of the US this week were the most bearish we’ve seen yet this year,” said Michael Waldron, analyst at Lehman Brothers. “But it didn’t really move the market.”

Analysts question whether other investment banks might collapse

  • 19/03 Icap gained on high volume trading but concerns that it is temporary
  • 30/03 Various – Lehman suing Marubeni $353m fraud and short selling rumours about Bear Stearns and Lehman Brothers on behalf of hedge funds.

Relationships:

Oil prices did increase above average for the month between 11/03 and 18/03 and then dropped 28/03 – 31/03. Lehman shares were more volatile between 14/03 – 20/03 then unexpectedly rose on 01/04. No mention of equities and other indices during this period so difficult to analyse any relationship even if ‘hinted’ but someone at the time, following the markets during this period would have had a better idea.

June

Market data:

  • 06/06 – Gold up (similar bigger jump to 15/05), oil bigger jump up, equities mixed, US rates down especially interest rates then by 10/06 oil down, gold biggest jump so far – down, interest rates up then towards 12/06 all mixed with another big but not biggest gold jump down
  • Lehman shares – 14/05 – 15/05 (as per big gold jump), open 43.77 closing 44.77. Then there was a gradual decline until June except 04/06. 06/06 open 33.22, closing 32.29, 09/06 open 30.01, closing 29.48

News:

  • 29/05 “Investor concerns about the potential negative impact of further equity issuance on the markets are misplaced, says Ian Scott, strategist at Lehman Brothers. First, Mr Scott says, issuance activity has been subdued in recent years. Even so far in 2008 the balance between equity issued and buybacks announced is close to zero. Second, he says, the unprecedented contraction in the global equity market in recent years means that investors have been able to build up huge portfolios of money market funds and bank deposits. “So there is plenty of scope for issuance to increase further, without damaging the medium-term prospects for equities,” Mr Scott says. Nowhere is the benign balance between issuance and buybacks more evident than in Japan, he says. “So far this year, buyback announcements have exceeded issuance there by a record amount. We remain overweight.”
  • 06/06 Attempting to raise $6bn having estimated loss of $3bn in second quarter

Relationships:

None  - equities remained mixed throughout June dates, no mention of commodities such as oil and gold swings in the articles in the selected sample*

*Approx 350 articles out of 70,000+ chosen due to mention of Lehman’s and sorted by relevance then date only. The sample does not include Factiva articles due to volume / available (human!) time.

 September

Market data:

  • 28/08 – mixed, some gold & oil slightly up then 01 & 02/09 gold (2 bigger jumps) and oil down, other indices mixed. 05/09 gold up by ½ of day before drop, equities down, 08 & 09/09 mixed across all indices.
  • Lehman Shares – mixed but continual decline towards September. 28/08 & 29/08 open 15.10, closing 16.09. 08/09 open 17.62 closing 14.15, 09/09 open 12.92, closing 7.79 before weekend.

News:

  • 27/08 – feature on Lehman commercial real estate assets such as NY HQ building in a healthy commercial property market
  • 02/09 – Reports of Korea Development Bank  considering 25% stake in Lehman
  • 04/09 – Sterling loss, oil fall – Lehman predict end of year prices for both
  • 07/09 – More on KDB and Lehman possible deal, brief review of Royal Bank of Canada July decision
  • 08/09 – 09/09 – Lehman share 52% decline and S & P warning to downgrade Lehman A rating due to uncertainty regarding Lehman’s ability to raise capital
  • 10/09 – KDB ending talks with Lehman

Relationships:

Lehman shares rose briefly between 02/09 and 04/09 probably due to KDB reports. Gold and oil dropped between these periods (don’t know if that means something or not). All market indices mixed on 08/09 onwards as reflected in wider press, also effect of S& P warning to cause over 50% share value decline.

Keywords March – June:

Difficult to suggest relationships without comparing to a month of relatively low financial activity such as a random month a few years ago or something like that, but for now:

March – highest frequencies include – fall/fallen (7), recession (3), shock (2), decline (2), crisis (2), fear (2), higher (2), better (2) plunge (2) reassure (2).

Others with only (1) but more descriptive include – anxiously, chaos, dropped, downgraded, engulfed, jitters, knocked, lower / lowest, nail-biting, predicted, perversely, plummeted, raging, rocketing, rushed, rumours, seesawed, sharp, speculation, spiral, struggling, tumbled, turmoil, unravelling, wiping, worried, worst

June – highest frequencies include –  leverage (16), loss (15) / lost (3), hedging (11), raise (9), likely (5), risk (5) / riskier (2) / risky (1),  expected (5), reduce (4), borrow (4), caused (3), fate (3), failure (3) declined (3), future (3), higher (3), fears (2), downgraded (2), harder (2), deeper (2), risen (2), opportunities (2), significant (2), stable (2), confidence (2), shackles (2), unwanted (2), worse-than expected (1) / worse (1) / worst (1).

Others with only (1) but more descriptive include – aggressively, avoid, awry, capsized, challenging, implosion, killing, negative, plunged, predicted, problem / problematic, radical, reduction, sank, self-confident, shrinking, spiked, stressed, tight / tightens, thornier, threat, toxic, tumbled, turmoil,  upgraded, vulnerable, wriggling, write-downs.

September – highest frequencies include – raise (8),  fell ( 8 ) / fall (1),  expected (5), declined (4), continue (4), potential (4), problems (4) / problematic (1),  state-run (4), lone (3), led (3), worried (3),  bad (2), cautious (2),  collapse (2), crisis (2), doubt (2), drop (2), fears (2), forward (2), future (2), gain (2), heavily (2), hedge (2), losses (2), privatise (2),  quit (2), rallied (2), rescue (2), rumours (2), runs (2), shake-up (2), strain (2), struggle (2), suffered (2), takeover (2), troubled (2), uncertainty (2), write-downs (2), worse (1) / worst (1).

Others with only (1) but more descriptive include – balking, downgrade, downturn, exploited, poor, predicted, stressed.

 STILL TO DO – time of entries against all data above, visualization post, economic analysis and anything else I’ve forgotten

Market data – first look Mar-Sept

The below is my highly scientific summary of the data. Failed my own data visualisation test because found it easier to look at the dataset instead of the visualisations – the visualisations give good instant indications but when trying to get per date – have to click on visualisation, scan / mouseover it to then get the information. At first glance – without looking at news, there are not huge significant swings overall in the 3 day periods but the overall data is quite limited for these days, it does not include every index in the overall tables. Decided that would also be useful to get Lehman shares data for the whole period.

Did this from Yahoo Finance – from historical data – it was really easy, I’m not sure what I was doing before, because I could have got 3 day periods from here – and it downloads as a csv ! (Will have to check re indices but it looks like you can). Is in my topic hub in Many Eyes as a scatterplot and dataset/visualisation on wikified dashboard.

 


 

11/03/08 Equities indices all up,  some commodities and couple of yen currency indices up, interest rates up

18/03/08 Gold down against $ Oil up against $, equities, currencies all up, interest rates – most up

25/03/08 Dow Jones –  slightly down, some equities & commodities up – less so than 18/03,  some interest rates up

28/03/08 – most indices slightly down except interest rates – some up

31/03/08 – Oil against $ down, gold against $ up, some equities and currencies down, some interest rates down

02/05/08 – Oil up, gold up, some equities and yen currencies up,  some interest rates up

09/05/08 – Hang Seng, Euro & FTSE up,  some others down, gold down, currencies – no data

14/05/08 – Hang Seng, Euro & FTSE down, some others up, most interest rates up

15/05/08 – Gold up (bigger jump) 1x Oil down, some other equities up, some interest rates up, some down

05/06/08 – Gold down, 1 x Oil up, 1 x Oil down, some other equities up, interest rates mostly up

06/06/08 – Gold up (similar jump to 15/05), Oil x 2 both up (bigger jump) most equities down, US rates down, 

09/06/08 – Oil x 2 down, some equities up, some down, interest rates mixed

10/06/08 – Oil x 1 down, Gold down (biggest jump so far), Nikkei down, interest rates up

11/06/08 – Oil x1 up, Gold up (c. 8), interest rates up, some equities down

12/06/08 – equities – some up, some down, 1 x Oil slightly up, Gold down (c. 18), interest rates – some up, some down

28/08/08 – 29/08/08 – equities – some up, some down,  interest rates, some up/down, Gold slightly up, Oil x1 slightly up

01/09/08 – Gold down (c. 16), 2 x Oil down, some equities down

02/09/08 – Gold down (c.25), 1 x Oil down, equities mixed

05/09/08 – Gold up (c.14),  equities down

08/09/08 – 09/09/08 – Gold up (c. 0.4), some equities up

Samples available for analysis

Up on wikified – on Sample 1, 2, 3

Also on Many Eyes on my topic hub of Sample Financial test 08

See also samples from tagcrowd which shows frequencies and includes filtered keywords using Lehman custom stoplist (i.e. not lots of Lehman and Mr etc)

Mar
Jun
Sep 

Am hoping to do some analysis tonight or tomorrow. The scatterplot / line graphs are not showing distribution around the mean (i.e. from what I understand of economics and standard deviation, distribution of the points in relation to the mean average can assist in helping to analyse possible market volatility)

On a separate note, going to show – whatever this is, to a lecturer in financial management at Surrey next week, so maybe it can go somewhere somehow…

Market data now up on Many Eyes – FINALLY!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Also (datasets+visualisations) on my regular Many Eyes page as well as wikified overview page.  The Overview page now has 10 different visualisations on it – as snapshots but still – 10 on a wikipage – completely amazing !!!!!!!!!!!!!!

Will find a way of moving / copying data from overview page with other visualisations into regular Many Eye too. There are 4 tests – each one is a comparison treemap – yay – it finally worked or I’m slowly progressing from total data visualisation failure to  heading in the direction of mildly competent. This involved a lot of reordering and messing with the spreadsheets but can’t think of a better way of comparing at a glance. If go down the ‘line graph’ route, then probably would have to take each index separately. If was only going to do one 3-days batch in one go, might choose different one, but because of trying to speed up – wanted to put all dates of market data (none available for 2007, but 01/08 – 09/08 there).

Will now look at for relationships, all visualisations now in for financial trends, the only one I might change is popularity of news items because it looks weird on the matrix graph, but others are done – for now.

Financial Trends software, starting to pull it together and more corporate thoughts

I think maybe due to the amount of text data, unable to save table pages in wikified at the moment relating to the news, activity and advice, but have pulled all into one table and is a dataset on regular ManyEyes.  Re keywords, wordle, word frequencies, (for wordtree – currently has investment as example), all data now in wikified. Couldn’t actually use the stoplist in tagcrowd in the end to import back in – because it doesn’t show frequency of words once sorted i.e doesn’t show a particular word 30 times or whatever, just has 30 next to the word as an overall figure.

So pasted from pdf – into odt document, used ctrl+f to find similar words to the stoplist and as many common words as I could think of, and replaced with blank space – then pasted that up into wikified. Not perfect but does the job as a test for now and the 4 different visualisations will help provide the overall picture of words. Unfortunately nothing interesting immediately obvious re word choices, style – would probably need smaller more detailed examples to analyse – but works in theory :-)

Task of the weekend is to match the dataset info above to market data and find a way of making the market data as in spreadsheet in yesterday’s post – viewable in an easy to understand, quick to grasp kind of way. Based on everything so far it doesn’t appear that there are any relationships immediately obvious, but to be honest, the market data is overall data (would be more useful to drill down into further indices and company share prices in relation to their mention in specific items of news) and the amount of news articles is probably too many – but again as proof of concept, it has potential – maybe?

Whilst I have been wading through various different types of financial data and visualisations on the web, it also appears that audio, video, text, pictures can potentially create more confusion and distraction (will post on this in more detail later). This is one of the reasons I have found Many Eyes particularly helpful for this – it keeps it simple.

There are lots of pieces of  financial data software (extraction, aggregation, visualisation) out there which require payment, Lee’s papers require organisational / individual IEEE subscription – so both of these options suck, in terms of everyone being able to access ! Corporately, both publishing of information and use of software make it more complicated – if this software is only being used and accessed by those who can afford (minority?), then what does that say about the reliability of data that both they are displaying which is hidden and the reliability of the data that the rest of the world sees ?  Or has the web made a difference and making some of this software less useful ?

Financial visualisation UPDATED

UPDATED On a separate note, now have all the data in ods sheets, now have to think of ways of simplifying - it is showing data over periods of time, so it makes sense that it would be some kind of line – not sure about stack graphs – because there aren’t really categories which would make any meaningful sense to a reader. It is too complex at the moment to make any sense to an online chart creator in terms of understanding what column needs to be represented where.  

Just spent couple of hours talking with Lee and Anna about computing ethics and financial visualisation  / risk analysis software – as to whether it was an idea even worth considering. Answer = yes but they don’t think ethics research is available to study in terms of funding academic research, based on Anna’s experience most people studying computing and machine ethics are doing it ‘for fun’ at the moment.

Have also found out that Lee has done some infovis with financial data so he has recommended some resources to look at – as per my recent delicious ones today, still got some more to add (menu bottom left).  He suggested Yahoo finance too re market data so I will go back and take a second look, couldn’t locate 3 days worth before.  He is interested -in wiki visualisation – beyond financial visualisation too – it is relevant to our overall Surrey research issue (more on that in a future post, may possibly be making some progress with oer / creative commons awareness building).

Although all the financial visualisation (i.e the time it will take me to go through everything) is probably beyond the scope of this test.

Market data madness 3 – v.short update

Have started to look at the suggestions kindly provided by Mr Flip @ infochimps – I LOVE their site and there are datasets there which will be useful for the future, going to review in detail when reach the end of this. I have been unable to figure out the example from the Edward Betts site, but will do.  The opentick one is not accepting new registrations at the moment, but useful for the future anyway. Mechanical Turk also not accepting requesters outside US at the moment.

Will also look at whole data extraction in more detail once through this. Not happy at all about my approach but at risk of sounding like stuck record, time is problem and getting data, converting to PDF then converting into spreadsheet is not easy. Found and currently using a 7 day trial version of Able2Extract (around $150 to buy in full), don’t know how many can do – but it did one ok, by selecting perform image-base conversion then left clicking at very bottom of pdf and slowly selecting all by going right-left-top ! Its going to convert them into separate ods sheets but can paste into overall one.

Converted into spreadsheet

If had more time would have searched for a better and usable solution but would rather just test out with small sample for now and figure out later.  All data uploaded via my dashboard and visualisations due to be created on overview page – however they are available if anyone wants to mess around with.

Going to take some time, hopefully can do the remaining ones this week at some point.

Market data madness 2…how to get from PDF and how to display

Ok, so I do taekwondo – we must learn perseverance and challenges must not be avoided in order to improve ourselves, so I am actively welcome this challenge with great joy, knowing that I will be a better person once I have found a way. Again, time is issue so will have to come up with interim – something.

At the moment even though I am having to do this manually – still can’t think of sensible way to display it all i.e. just on Equities summary data, you have various indices, so adding 3-4 dates worth of data seems impossible to display in a simple table format, as per below. I tried to display this as a visualisation in Many Eyes via a line graph (it even suggests using a line graph for stock prices) but it wouldn’t save it. I tried in regular Many Eyes and it saved ok. So maybe its a wiki test thing.

714ebc72-e001-11dd-a102-000255111976 Blog_this_caption

I tried this using the chart tool in open office spreadsheet and in one second it had produced this – I didn’t select line graph – could have done, which is also fine: 

screenshot from openoffice spreadsheet

screenshot from openoffice spreadsheet

Market data madness – if only I could write APIs……………..

UPDATE – managed to get market data from Yahoo Finance but as individual indices not as overall tables as per the FT archive.  This is free but time consuming. however, can paste it into Google Spreadsheet and set the publishing to autoupdate.

You can use fee-based data subscriptions such as DataStream but this is for institutions, very expensive for individual research.The Open Economics datastore also has some free options.

 


Too frustrating. FT has archives e.g. data available only from Sept 07 onwards – for sake of time chosen summary data per markets e.g. market summary equities, market summary commodities – rather than go each set of indices. This kind of a task needs someone who understands and can write an API to pull it (seeing as you can only search by month and it will display one month at a time. I haven’t the faintest idea how you can ‘get’ this data out as an overall set *. Also don’t know if you can see this data without a subscription, I’m logged in at the moment. Due to this, only doing this based on key dates between 09/07 – 09/08 as in the history overview from previous post.

 

( If I was doing this as a project – scope creep would have officially gone through the roof and out into the atmosphere – everything is taking so much longer than I estimated due to complete lack of automation in getting the data.)

The Economist does not appear to have an archive where I can quickly get this data, it has an amazing archive of every single one of its print editions, similar for Times and Telegraph – but can you imagine having to go through each one individually – total insanity. The Market Data Centre website requires subscription and based on its demo, I think I could get weekly info but not the pattern of 3 days before a particular date which is what we are trying to do for this financial trends visualisation. There are multiple chart tools on various trading tools websites, again subscription required. Factiva doesn’t have data at least not in the bit I subscribed to. We definitely need more openness in the financial world, not everything should be subscription based, there must be an alternative revenue model.

BUT – this is a major pain for me but would be less of one for anyone in the future unless they were doing a look back over a similar period. If someone was just doing one set of 3 days from an event that happened recently, it wouldn’t be so much data to analyse.

* If I had spent more time since my CIW doing web work then I might be in a position to write something – but don’t have time to learn / relearn anything right now – assuming its some form of SQL queries that would need to write (only ever wrote very basic ones and that was 3+ years ago – haven’t done since). Sweet Tools or other data sites such as theinfo might have something – but the amount of time it would take me to search and even if I could, locate something suitable would probably be as much time as it is taking me to do this very long-winded way. Unfortunately, I don’t have a fellow web person to hand who would donate their time for free – and no one responded to an earlier wikis and visualisation request I sent out earlier this year.

Going slightly off track on this but worth an explanation – a lot of the wiki work I’m involved with at Surrey is on the research side – they are putting in multi-million pound bids for research funding and keen to find better ways of collaborating. The first time I became involved was when I was asked by someone I didn’t know at the time to come and explain about wikis with about 2 hrs notice (i.e “oh the bid deadline is in about 6 hrs…”)  luckily this went well and as a result of this, each time a bid comes up – I get asked usually with similar kind of notice to go and talk about social networks, wikis or whatever and they add me into the bid.      There is so much work with wikis and other web tools that could create and display visualisations, that can benefit Surrey – the thing is they have computing science, cognitive science, Digital World Research Centre and lots of other places – so they have both technical and design experience but nothing is joined up and culturally too, there is a certain amount of misunderstanding, fear about what can be shared or not. So mini-projects like this are a great way of starting these conversations with different people. However it really needs someone with considerably more expertise than me.   

 

Visualisation of history overview (this is boring 1 paragraph not worth reading)

Uploaded into Many Eyes as table with 2 columns just date and 2nd column – 1 line of key event in their history. Need to go back and reformat because have mix of years and dates with exact day/month/year – knew this wouldn’t work but just wanted to get a feel for what I could do. The selected visualisation was a block histogram but it doesn’t have enough data in it – tried something more as a line i.e. line graph and that kind of worked better but it treated dates as numbers (because I had specified it to do that otherwise it wouldn’t output a line graph) but I had a problem saving (not surprising) so will return to – whenever. Ideal visualisation for me would be something with a line with key events but can’t think of anything more specific at this point and need to rework data in order to do that.

Keyword filtering in Many Eyes – quick test

Did some quick tests last night – wordle & tagcloud – on wordle one I could specify up to million words but it took a long time (unsurprisingly) and results mostly unreadable. I then chose maximum of 1000 words. It was ok – for this purpose. I’m also not entirely sure it picked up all the article content in the ‘data’ page, I pasted it in from a 96 page pdf and it looks ‘short’ but it might be all there (Note – check this) With tagcloud, it gives option to tick if you want to filter out common words like and, the but you can’t specify individual words so both visualisations had the most common things like Lehman, finance, markets which are words that I would choose to filter out. So I created a custom stoplist on Tagcrowd (its public – just go to the stoplist dropdown arrow, its called Lehman – at the moment it only has about 8 words in it) and clicked visualize ! Chose 1000 words as limit too so it was unable to display on one page.

View / download

Need to think about how to get this back into wikified – either find a tool that can filter keywords out quickly from a document or will have to do manually.  Or rework content ! At quick glance it doesn’t look like there is anything was hoping to find but that’s useful to know as well. Still haven’t finished reading through all the content yet so will need to change, this was just a quick test. Wordtree will be also useful for this because can take a word now like ‘markets’ and see what it is related to.  Realistically for a small sample test this would be fine – I am using content from entire period so could take e.g. 3 days and test and display results coherently.

Have removed password from mindmap too – still updating there – would like to to be able to map everything got on delicious onto there but it only allows 1 URL per strand !

Everything I think I want to think about (updated)

Mindmap

Its on a new URL sorry, I didn’t realise it changed it when I removed the password ! I couldn’t see any value in having one, don’t think anything really confidential there
Sure to have missed lots of things off, should have done this weeks ago.

RSS feeds

Just writing this down for reference because it is relevant to financial trends one and will come back to, whilst reading through the overall news entries.  We discussed use of RSS for ease of picking up news from source with date & time relevance. Request from SME included 

“Would it be possible to use one of the exchange feeds to act as a source which would capture (say) the individual movements over a period and would show what the market is trading (i.e. which securities are being a) traded most . This would give you an insight into real time development and presentation of themes, just the kind of thing that sometimes goes down well on trading floors (and they will be back) … and despite the papers, there’s still tons of money in the financial markets, even more if you can give someone a tiny edge….research to see if traders do better or worse with affiliated market data presented in different ways”

I haven’t really begun to think about yet but have started to put together feeds and thinking to pull with other data via Yahoo pipes as per this very early draft (via edit source) http://pipes.yahoo.com/pipes/pipe.info?_id=AMuxnvXc3RG3WjEzpgt1Yg
Related to the financial trends visualisation, if you could combine this with keywords then it might be possible to visualise market swings based on these news sources and financial blogs and also combine with a search engine in Yahoo Pipes to inform about any new occurrences of keywords etc.

The reason that I’ve added in keywords as one of the tabs on Financial Trends examples on Many Eyes is that I’m guessing that each newspaper must have some editorial guidelines for article content and presentation. So maybe it will be possible to spot keyword occurrences then if possible to also evaluate alongside style / tone type keyword differences (i.e. is there a difference in language as a result of one news story published which is mentioning a possible incident then when it publishes a related item after this, has the language changed in any way). It would then  be interesting to see if there is any relationship between any of this (especially if time of item available) and the actual market data at that time. Not thinking of a wordle or tag cloud type word visualisation more something along words floating along lines like the lines in this visualisation
You could then filter by date or time.

So going back to the overall possibility of an exchange RSS feed – combining the above in maybe Yahoo Pipes, creating a visualisation then maybe a second separate pipe which shows keywords and creating a visualisation of words over time ! Then combine all of this back into an exchange feed, i.e if it was on Many Eyes wiki, you could subscribe to different visualisations and combine the RSS from each separate visualisation back into a pipe.

Forgot to mention Factiva – providing you had the relevant subscription, adding these in too – e.g. matching times to the local entries in Factiva say from newspapers in Hong Kong or wherever, any evidence of relationship to the others newspapers above would be interesting – could visualise in same way as the above.

Time – less complex data but more complex visualisation that makes sense to me

The data is uploaded with time, date and publication title on Time.Entries

The visualisation also available as a snapshot on overview page and as a matrix chart. This is actually a non-complex set of data but its a slightly longer column than a ‘page view’ with 100+ rows. I think but again with limited understanding – this matrix chart works because I have the times set then can choose to filter by date or publication. Later on, when try and get hold of market data this will be useful for comparison – although the ideal then would be one which shows market data, time of news entry, publication and date all in a single visualisation..maybe (need to think about more).

Visualisation for Entries

This was always going to be a strange one because of the large amount of dates spread over the period, but was attempting to show number of entries from each news source for a series of dates in the sample between April07 – Oct08. Once created in a spreadsheet (as per earlier post, second time lucky), was simple enough to copy and paste into wiki page. When I clicked to create the visualisation, it chose a matrix chart as the best suggestion for displaying it.

It looks a bit weird – You can see the different number of entries when you filter by dates (if you use regular Many Eyes, the filters are at the bottom of the chart). I filtered by dates and selected bars instead of bubbles so I could see quickly where there were higher numbers (although having created the source data myself I knew that anyway and could guess that there were going to be a lot around mid Sept 08 !). I have tried using a different visualisation but it might mean going back to the drawing board with the data again and seeing if there is a better way to put into a table that might produce something more meaningful?

I can pick up the script ok but unable to embed in this blog at the moment, due to time have not looked at the settings in any detail, if can’t find a solution quickly will continue to embed on other site and link to from here.

Experiments to date

Firstly quick note about my main two SMEs for this – one of whom is my sister who is a statistics ‘expert’ of sorts.  On a personal note (so skip the next 3 paragraphs if you want to get to the real stuff), this has been really amazing connecting with my sister in this way, we hardly ever mention work and no real idea about the kinds of things that each of us do, I never imagined that we would collaborate – now we have been chatting about wikis, superb !

I mean she has an important job but very very occasionally in the past – I think the last time was back in 2000, I used to send her graphs I had made – very creative ones IMO – I used to obsess about the design of graphs – I hated seeing reports where all you got was a boring white background and those kind of  lilac purple sets of  bars, I used to spend hours (yes, really) turning mine into works of art – with photos and whatever kinds of colour/shape options I could find / come up with.  Anyway I would send these graphs where I was trying to present about a million different sets of data  all in one place and ask for expert opinion. Imagine if you are used to sending teams of researchers out to collect data and present reports then you get this ‘thing’ from someone who wouldn’t even make a junior researcher probably – hopefully you get the idea ! She is currently seconded to a different govt office but coincidently someone asked her about Many Eyes the other day in relation to maps, so the word is spreading :-)

My other SME (due to not getting hoped contact from Surrey Uni but I think is better anyway) has lived and breathed financial trading, in both senior financial and technology roles, so has been really helpful. I just by chance happened to suggest an idea about possible scenarios for investment, mortgage etc and he has been really helpful in bringing more specific detail and suggestions to this, that I would never have known myself. For example yesterday, he mentioned that in some ways he really hopes that we don’t find any relationship at all between the news and market trading data, to just provide evidence of how useful the science of prediction actually is or not, especially in the financial press ! And there is so much even in the current financial press this week about prediction or not !

Right so that’s them, onto experiments:

1. Financial Trends

Displaying as sub-blog due to volume of thoughts so hopefully can see processes / stumbling blocks etc

05/01/09

Data is uploaded as table and visualisation now available. All source data is now on / linked to from wiki. Spent 3 hours creating a table in a regular openoffice spreadsheet but due to own incompetence, still not quite in format needed for Many Eyes so now redone – interesting though, I think I’m starting to get it now – you are making the data less complex at stage 1 which is the table creation stage, then displaying it in a more visually appealing format is so much easier, however if it was a case of pulling something off the web, it would probably need to be edited down in the same way as above, difficult to embed directly. Table includes Economist, Times (up to 16/09/08), Telegraph, FT but not Factiva yet, need to point out that not actually read the articles in any detail yet so 50/50 about whether included right kinds of ones in sample (approx 350 – 400 out of several thousand), reading them is next task ! Am relying on current level of personal understanding of all things economic to hope that I have picked up the right ones, will soon find out or not. 

Other thoughts re visualisation and presentation which may be related:

http://www.fotosearch.com/photos-images/bar-chart.html i.e. this is just an example of one type of graph
Notes for ideas (also http://www.telegraph.co.uk/news/picturegalleries/worldnews/2971024/Stock-markets-around-the-world-fall-after-the-collapse-of-Lehman-Brothers.html ) Also FT ‘mouseover’ live company trading charts. (also Mind Hacks 19, 26, maybe 32, 37, 38, 39, 56, maybe 61, 82, maybe 85) Design does not equal – creating new tools or instruments – more subliminal for noticing trends(?), short-term and long-term processing of change visually and maybe other senses, creating relationships) Also Teemu’s post http://tarina.blogging.fi/2008/11/06/subliminal-pattern-recognition-and-rss-readers/

What about audio?
http://news.bbc.co.uk/1/hi/business/7688308.stm audio slideshow, http://www.fxdd.com/en/forex-trading-tools/trade-the-news.html audio broadcasts etc
how does audio help in terms of understanding financial complexity (Many Ears? ) need to investigate – was thinking along lines of – if you are already looking at a zillion different coloured things moving across your screen and there is something more subliminal in terms of capturing your attention and then directing your vision, would there be any benefit in a subliminal ‘whisper’ which could later change tone & volume – or maybe that just too creepy or weird, asked SME, he was slightly intrigued and prepared to consider possibility.

02/01/09

Have selected large investment bank as sample due to volume of news, size and reputation, impact on other economic areas. At the moment compiling entries from online versions of Economist, Times, Telegraph, FT (Factiva if time….) Overwhelmed ! Not sure at this point re wiki pages how to display and format e.g. have a spreadsheet with14 tabs and document – 50+ pages so have temporarily set up 14 sub pages from Dashboard but if can display more than one visualisation and more than one table on one wiki page, maybe (benefit or not?). Economist & Times show posting dates but not times, so idea of trying to look at relationship to live data may not be possible from those news sources, but Telegraph, FT and Factiva do show times. Blog / comments volume probably too big to do as sample but will try if time. RSS / pulling data options limited due to FT, Economist, Factiva requiring paid subscription to view either some or multiple articles during specified period.

23/12/08

Currently to develop visualisations of live market data compared to commentary in both financial and wider press online, also if possible against financial news aggregation tool. A third option would be to evaluate general company analysis and stock market tips. How to distinguish reliable information against volume generated and impact of speculation. In order to create realism to create a possible future scenario of individual events and how market responds, displaying as visualisations, evaluating Apr07-Oct08 of key events, not sure if RSS available within wikified , but regular wiki pages would be possible, so not an issue in principle for a ‘proof of visualisation concept’ test on a wiki, yet. Complex due to e.g. 700K+ entries in international financial press in one tool, in UK financial press also similar number of entries – for one particular company during this period. Once have an idea from this evaluation can attempt to generate for other financial events.

2. National Survey data

 Alcohol, child etc – these are already tables, could upload into general Many Eyes and link in – but non-live data. Inserting manually then as grid format or as data table does not seem to make difference, both take time manually and results are less ‘attractive’ than original table. Don’t have original datasets collected by researchers which would be more useful test for being able to visualise complex data. These are national surveys and the data process has already been completed for these reports by the researchers. Have checked other tables in case I was inputting DCMS tables in wrong format but have tried with just a sample of a table and still not appearing in same format as e.g. big leo table (in regular and wikified Many Eyes).

In this case, a survey process typically involves interviewer/researcher recording data electronically which is automatically fed into a statistical analysis tool which includes databases etc Then data can be pulled out by relevant parties as needed, most common way has been to capture and present in spreadsheet incl chart tool (is it minimum variables versus multiple – which is better). So someone would not be viewing a complex set of data, but possible that they might not be able to follow / understand data reports presented. If cannot find quick way of uploading into web spreadsheet or embedding data directly from spreadsheet then maybe not an advantage to Many Eyes but updating in real time would be very useful. 

Checked data upload process for Many Eyes - Tables have to be simplified in both general & wikified – either convert columns with multiple units of measurement (g, ml etc) into a column with just 1 unit or removal of ‘junk’ defined as subheadings, multiple unit columns, columns with mix of nos & text. Cut & paste from web page into text editor then, can paste into spreadsheet e.g.open-office, then reinsert meaningful header names (specifying measurement unit if applicable), then can paste into Many Eyes – it should recognise as table. Hmmm – with tables I’m using, can do some of this but what about the ‘junk’? Won’t this mean that only really basic info can be displayed so does a visualisation of colours/shapes add that much more value than seeing in a basic table? eg on CSC it will mean creating 3 or 4 separate tables out of one table too. Saves time, you can see at a glance, differences instead of scanning table rows (audience/context) How does it make better decision-making? Want to test table vs charts to see for myself. (Note: this was before reading Mind Hacks over Christmas, since reading that I have a better idea about this even though I haven’t tried the specific hacks yet).

Due to needing to think more about table formats (spent one session with sister with Many Eyes open but with small twins needing attention and the room being freezing, it wasn’t too productive, also due to the fact that I hadn’t really ‘got’ table format required) will return to this one financial tests completed.

3. Text analysis and visualisation.

2 tests – one basic one pulling the text (cut & paste) onto a page in Many Eyes then creating visualisation by clicking a button. Was able to embed snapshots of 3 different types of visualisation to compare – tagcloud, word frequency bubbles and wordtree which is brilliant, small snapshots of graphs on a wiki page which you can then open separately is brilliant for comparison. You could also do this now in regular Many Eyes and test it yourself by embedding three graphs on a web page if you know how to create a web page etc.

Wordtree example:

http://www.aydindesign.com/garden/complexcommentsmew.html

This is of complexworld homepage wiki comments (23 comments) – visualisation loaded ok, searched using “learning” as search term – 4 hits, “inquiry” 5 hits, “enquiry” 2 hits, “how” 8 hits, useful! Added bubblechart for word frequency in 23 comments – originally thought it would be easy to pull word frequencies using tag crowd, but was nightmare to import into a spreadsheet using import html or just cut & paste. Ended up selecting list from source code of tag crowd result page, pasted into Word, converted text to table then had to edit out the html – tedious. Would probably have been better just to paste all the comments into Excel then use frequency formulas to do word counts (will look at later). In wikified – tried importing again from google spreadsheet and many eyes dataset but worked better when pasted two columns directly into the wiki page. Bubble chart shows either one colour or multiple colours but colours don’t relate to word frequency differences. Really useful to compare against other ways of visualising comments such as word tree and regular tag cloud. Tag cloud is interactive within the page so you can go straight to where it was found. (todo-embed this in webpage)

The second test was similar  - a wordtree test for a group of people in computing / engineering faculty. They liked ease of use, presentation, asked for additional buttons and queries about frequency/occurrence difference. You could also do this with regular Many Eyes and it provides explanations for all the different types of visualisations there too.

  • Archives

  • July 2014
    M T W T F S S
    « Jan    
     123456
    78910111213
    14151617181920
    21222324252627
    28293031  
Follow

Get every new post delivered to your Inbox.