Data Mining

Average SeaWiFS-derived chlorophyll-a for southern hemisphere summer 1998/99 to 2004/05.

What is data mining?

Data mining:

• is the process of finding new information from existing collections of data
• draws on a variety of tools and techniques
• uses data from many different sources in order to exploit relationships between those data.

Why are we interested in data mining?

Collecting data from polar regions is difficult and expensive. Through data mining, the Data Centre seeks to make the data that we hold more useful to the Antarctic community.

Data mining is used to enhance the value of the data held by the Data Centre in several ways:

• analyses of multiple data sets to uncover information which would not be obvious from single-data set analyses
• using exploratory techniques to uncover new information from the data
• producing actionable information and "end product" data from low-level scientific data
• to generate a better understanding of the holdings of the Data Centre. This includes data management information, such as:
  • data errors, and duplicated or missing information
  • linkages between databases
  • insights into ways of improving our data acquisition procedures.

Data Mining Projects

Current data mining projects

• A large-scale model of sea ice algae development in East Antarctica
• Investigation of interannual variations in ice-edge chlorophyll blooms in East Antarctica
• Network-based visualisation and exploration of scientific data
• Network-based methods for community ecology
• Web-based animations of ARGOS satellite tracks (animal movements) with physical environmental data such as sea ice
• Involvement in marine bioregionalisation exercises, in the Southern and global oceans

Past data mining projects

• Observations of seabirds at sea
• State of the Environment Reporting
• Gas chromatograph analysis
• Leopard seals on Macquarie Island
• Computer-assisted discovery of scientific knowledge

Publications

Request a copy of any of these publications.

Journal and conference publications

Draft or in-press publications

1. Raymond, B. and Hosie, G. (2008) Network-based exploration and visualisation of ecological data. Submitted
2. Schwarz, J.N., Raymond, B., Williams, G., Marsland, S., Pasquer, B., Mongin, M., and Wright, S. (2008) Climatological anomalies in wind, sea surface temperature, sea-ice and chlorophyll concentrations during the BROKE-West survey. Submitted
3. Woehler, E.J., Raymond, B., Boyle, A., and Stafford, A. (2008) The role of environmental determinants on seabird assemblages observed during BROKE West, January - March 2006. Submitted

Software

The Polar Toolbox is a collection of Matlab functions for various analyses of Antarctic data, but which might be useful for analyses of other data. Most of this code has been tested under Matlab 2008a. It may also be compatible with other applications, such as Octave, which is a freely-available, mostly-Matlab-compatible package.

All code here is experimental and made freely available. Before downloading you will need to register (free!) with the Australian Antarctic Data Centre. This allows us to direct our development attention to the most active files.

Download the toolbox

Queries?

Toolbox contents

% Polar Toolbox: A Matlab toolbox for various Antarctic analytical tasks
% Ben Raymond 
% May 2008
%
% aloc - Belbin's ALOC non-hierarchical clustering algorithm
% cellmax - the maxima of a cell vector of identically-sized matrices
% cellmean - the mean of a cell vector of identically-sized matrices
% cellstd - the SD of a cell vector of identically-sized matrices
% cluster_clara - clustering using R's CLARA function. Requires R and the matlabRlink toolbox to be installed
% cutree - cuts a dendrogram and return the group membership vector 
% dump_csv - write numeric matrix to comma separated text file
% easegrid_ll2rc - transforms latitude and longitude coordinates to row and column coordinates of an EASE ice data grid cell
% gebco_colormap - colormap similar to that used on GEBCO bathymetric charts
% get_ice - retrieves ice data from binary files as obtained from NSIDC/seaice.de/etc
% get_icemotion - retrieves ice motion data from binary files as obtained from NSIDC
% icemotion_easegrid_ll2rc - converts longitude/latitude coordinates to row/column coordinates in the ice motion EASE grid
% ndvi_colormap - colormap similar to that used to show normalised vegatation index images
% proplim - confidence interval of a proportion m/N
% ranks - ranks of x adjusted for ties
% slidingmean - applies a sliding mean to a vector
% slidingmedian - applies a sliding median to a vector
% spearman - computes Spearman's rank-order correlation coefficient for two vectors x and y
% sunriseset - calculates sunrise and sunset times