The Internet has made it easier to donate to presidential campaigns, so much so that the Federal Election Commission has had a hard time keeping up with the seemingly sudden influx of data they have to process.
The campaign finance reports filed by Obama and Clinton have grown so massive that they’ve strained the capacity of the Federal Election Commission, good government groups, the media and even software applications to process and make sense of the data.
Hold up. Even computers are buckling under the pressure? The first things that came to mind were crashing servers and tech maintenance pulling their graying hair out. Reading on though, “software” is a reference to Microsoft Excel 2003, which can’t handle data files larger than 65,536 rows or 256 columns.
Phew, that was close. I mean, come on, this is nation-wide data. Give me a MySQL dump for Pete’s sake.
Anyways, tt’s certainly a good indicator for how times have changed data-wise. Excel 2007 can handle more. And on that note – it’s still possible to open John McCain’s monthly reports in Excel 2003.
[Thanks, David]
Time to trash Excel and Microsoft again. People are using a spreadsheet application as a database, and running into problems! Why couldn’t Excel work like a database, instead of a spreadsheet? Obviously a flaw in Microsoft’s design, not in our usage of the tools.
At least if you upgrade to Excel 2007, you’ll have millions of rows, plenty of room. Never mind how slowly it will run, you get special visual effects which are so kewl, d00d
Why title it “Excel cant handle … ” ?? Isnt it better to be honest and say Excell 2003 cant handle some data? I mean, what will people do next to get their blog readership up.
@Jon: as i was writing this post, i thought of you. i was expecting a slightly different reaction :)
@aj: a valid question, but like jon said, when you’re dealing with a spreadsheet with hundreds of thousands of rows, Excel will work in the sense that it’ll load, but the usability is just painful. So while excel can “handle” large datasets, it can’t really.
This raises the question of why the FEC is using Excel 2K3 as its main computing tool, rather than an SQL flavor. Don’t get me wrong, Excel is great and I use it for all sorts of things, including things that would be better on a database, because I’m more comfortable with Excel. But this is a federal agency with a budget and the ability to put the data in the proper container, not a once and future journalist in his home office with a cobbled-together collection of outdated commercial and more recent free software.
@Nathan: What response had you expected? Excel isn’t a database (though it’s great for analyzing databases), and people are blaming Excel for having this shortcoming, rather than using the proper tool for the job (the proper spanner to pound in the bolt?).
@Pierce: So why not use Excel 2003 and SQL to analyze the database? Excel makes a decent front end. I’ve used Excel/SQL to analyze Access databases, all manner of text dumps, and even other Excel workbooks. Filter the data before it even gets into Excel, and it will go much faster.
@Jon – Unfortunately Excel 2007 can’t handle ‘millions of rows’, but rather about 1 million. Yes – I’ve bumped into this limit more than once.
Jon’s suggestion to use Excel in conjunction with a database sounds interesting… Any pointers to tutorials on how to do this?
To do the Excel/SQL thing, just look at the Data tab and the “From Other Sources” dropdown. It’ll bring up an ODBC dialog, and you can use that to point to an SQL table.
I tend to create specialized views that reflect what I want to show inside Excel so that I don’t need to filter inside Excel.
(This also works wonderfully with PivotCharts.)