Well, it seems I’ve managed to sort out that data parser I was working on – it finished its first successful run at 2.34am this morning. However, I was just falling asleep before it completed as I’ve just been too tired to do much (amount of code added to the parser since last week: 6 lines of regexp’s – number of runs to attempt to parse the data – I’ve lost count!).
So now I’ve got 1Gb of SQL statements that I’ve got to load into a database whilst combining at least another 85Mb of unstructured data to it. Then I’ve got to parse out HTML/PHP pages which has dynamic remote XML/SOAP parsing systems with cachability whilst ensuring the whole database can be searched – and then I’ve got to work on stage two which is “data expansion” as then I’ll have enough data to seed the new system which will use the dynamic SOAP system to capture even more data whilst another system spiders a few items. It’s hard work (even the theoretical planning and timeframes a beast), but hopefully I’ve have a helluva good system by the month end if I can actually get time to work on it.
Why don’t I devote more time to it? Well, let’s just say this – I’ve been at work 3 days so far this week and yet I’ve spent so long in the office I’ve already done 4 days work if you count the hours (Monday was systems upgrades and hardware modifications to the sales machines, Tuesday was finishing off the coding for a dynamic employment site along with finalising some bits with two new billing systems/CRM systems I’ve written, today was “manual labour” – move lots of stuff half way across the building and up a floor and then totally rearrange and rewire the office). And tonight I need an early night as I’m going “preliminary job hunting” tomorrow.