AstroGrid: In the days before 'Big Data' there was Astronomical Data [Symbol]

Legacy photographic prints were being digitised (eg SuperCOSMOS ). Modern sky surveys were scanning the universe at increasing resolution and sensitivity (eg Sloan Sky Survey ). Bespoke processors sucked these pixel images into a pipeline that cleaned them of satellite and camera artefacts and extracted characteristics of stars and galaxies and nebulae. Such pipelines were highly sensitive to configurations and would be re-run with different properties for different end analysis

Data transfer often had to be physical: data was copied to a disk drive and then the drive or complete PC parcel-posted to international collaborators.

As instruments improved and proliferated, data size and complexity rocketed. Improved internet bandwidth trailed in relative terms. The data was not just 'big' but rapidly growing, poorly described, and difficult to associate.

The International Virtual Observatory funded several research streams to investigate software solutions. AstroGrid was the UK contribution; a set of web services organised in a 'grid' that could accept queries, manage the distribution of sub-queries and processing, and return answer sets. The grid service moved the task to the data instead of moving the data to the analyst.

Martin Hill (linky to staff pages) was responsible for the data publishing, archiving and querying services, interfacing and integrating with other services and providing a set of libraries to help others build their own. He contributed with many others to modern services such as the Aladin Sky Atlas .