tag:blogger.com,1999:blog-5467119.post7776619124299369958..comments2023-10-24T10:09:22.146+01:00Comments on The Yorkshire Ranter: lobby: updateAlexhttp://www.blogger.com/profile/17153530634675543954noreply@blogger.comBlogger7125tag:blogger.com,1999:blog-5467119.post-86155949351946253602011-04-20T12:58:47.959+01:002011-04-20T12:58:47.959+01:00The categorical nature (dividing lobbyists by indu...The categorical nature (dividing lobbyists by industrial sector) of the data makes Correspondence Analysis particularly applicable. As well there are a bunch of machine learning methods that would work nicely. There's a standard set of bioinformatics data exploration tools I'd like to try throwing the data at to see what happens.<br /><br />As for visualization, there are some pretty pictures that fall out of PCA and correspondence analysis.<br /><br />My problem with large network graphs is they rapidly turn into "ridiculograms", uninformatively dense meshes. This is common with protein-protein interaction graphs where there is a lot of noise in the data.<br /><br />Will download the data, please post when it's tidied; long weekend coming up to play with it!theseedmasterhttps://www.blogger.com/profile/15259531684267695761noreply@blogger.comtag:blogger.com,1999:blog-5467119.post-2625834094203642142011-04-19T22:39:15.187+01:002011-04-19T22:39:15.187+01:00Gawp: PCA is a cool approach. NetworkX's clust...Gawp: PCA is a cool approach. NetworkX's clustering algorithms might provide something similar (<a href="http://networkx.lanl.gov/reference/algorithms.html" rel="nofollow">list</a>). As far as visualisation goes, I've been looking at strategies - I like the idea of a radial-graph look, and the CAIDA Skitter graphs are an inspiration but that does mean reimplementing it for an undirected graph.<br /><br />You can get the data out of ScraperWiki, but I need to backport some data cleaning from the project into the scraper.Alexhttps://www.blogger.com/profile/17153530634675543954noreply@blogger.comtag:blogger.com,1999:blog-5467119.post-91287038401571986582011-04-19T22:24:30.819+01:002011-04-19T22:24:30.819+01:00Anon: AppEngine doesn't have this lib: Network...Anon: AppEngine doesn't have this lib: <a href="http://networkx.lanl.gov/" rel="nofollow">NetworkX</a>.Alexhttps://www.blogger.com/profile/17153530634675543954noreply@blogger.comtag:blogger.com,1999:blog-5467119.post-36864860166525502442011-04-18T14:18:10.876+01:002011-04-18T14:18:10.876+01:00You can generate a set of weighted values by lobby...You can generate a set of weighted values by lobbyist for ministries they lobby. This would allow clustering of lobbyists by activity; banks should cluster together, for example. <br /><br />Principal Component Analysis on this would probably be best for this, as you would be able to cluster by direction in lobbyspace verses position, this normalizes for intensity of lobbying. This will show what percentage of lobbying effort is explained by the first 2 PCA vectors; this might lead to some nice clustering right there. If, say, 50% of lobbying is on the first vector investigation of that vector will tell you a lot about standard distribution of lobbying efforts. Analysis by sector will tell you something too. <br /><br />And similar analysis on the ministries might work too; a weighted vector for each ministry of who is lobbying them. What ministries are most similarly lobbied? PCA of this would show what spectrum of ministries is lobbied and by what proportion.<br /><br />Much of this is probably predictable, but it will give a nice visualization and anomalies are often informative.<br /><br />Do you have a data set handy? I'd love to have a go at it...gawphttp://gpalidwor@yahoo.comnoreply@blogger.comtag:blogger.com,1999:blog-5467119.post-6256333549732997282011-04-18T10:58:57.090+01:002011-04-18T10:58:57.090+01:00Rather than using on off line cron job and uploadi...Rather than using on off line cron job and uploading, why not use the new pipeline library to massage all the data on appengine:<br /><br />http://goo.gl/MDrE5<br /><br />Have you published the source for this somewhere?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5467119.post-40398451185402000502011-04-17T18:38:43.571+01:002011-04-17T18:38:43.571+01:00Because Scraperwiki is for scraping.Because Scraperwiki is for scraping.Alexhttps://www.blogger.com/profile/17153530634675543954noreply@blogger.comtag:blogger.com,1999:blog-5467119.post-34437271313128082912011-04-17T18:35:58.948+01:002011-04-17T18:35:58.948+01:00What's the thinking behind taking it off scrap...What's the thinking behind taking it off scraperwiki?Lord Blaggerhttps://www.blogger.com/profile/06783119146180259097noreply@blogger.com