Many organisations (from the FAO, to the OECD, to Twitter or the World Bank, for example) make vast amounts of data available through application programming interfaces (“APIs”) that allow for the compilation of datasets for any kind of purpose. Once you have identified potentially valuable datasets, the generation of databases to include in your own applications can be automated to make sure you always have the latest available data at your disposal.
Often times, data is freely available on websites, but in human readable formats rather than machine readable formats. Such data can nevertheless be compiled into useful databases if their pulication is systematic enough to warrant “scraping”. For example, government organs may publish regular updates about work programmes or contact information, which could be automatically “scraped” and fed into an always up-to-date database.
Once you have generated databases, these can be integrated for use in different ways, whether as Excel sheets for people to use as they see fit, or in an appropriate format for other programmes or apps to redeploy according to their defined needs (see also automated monitoring, automated data analysis, automated data visualisation, automated publishing, automated reporting).
In brief, we will use the data you send us only to fulfil your requests, we will do our best to keep your data safe and we will not transfer it to anyone else, unless required by law or necessary to fulfil your requests.