Data Mining

Community structure graph derived from observations of seabirds and whales in the north Atlantic. Read more …

What is data mining?

Data mining:

• is the process of finding new information from existing collections of data
• draws on a variety of tools and techniques
• uses data from many different sources in order to exploit relationships between those data.

Why are we interested in data mining?

Collecting data from polar regions is difficult and expensive. Through data mining, the Data Centre seeks to make the data that we hold more useful to the Antarctic community.

Data mining is used to enhance the value of the data held by the Data Centre in several ways:

• analyses of multiple data sets to uncover information which would not be obvious from single-data set analyses
• using exploratory techniques to uncover new information from the data
• producing actionable information and "end product" data from low-level scientific data
• to generate a better understanding of the holdings of the Data Centre. This includes data management information, such as:
  • data errors, and duplicated or missing information
  • linkages between databases
  • insights into ways of improving our data acquisition procedures.

Data Mining Projects

Current data mining projects

• Visualising field trips and track usage on Macquarie Island
• Using the Wii Remote for data exploration with Matlab
• A large-scale model of cumulative light exposure of sea ice in East Antarctica
• Investigation of interannual variations in ice-edge chlorophyll blooms in East Antarctica
• Network-based visualisation and exploration of scientific data
• Network-based methods for community ecology
• Web-based animations of ARGOS satellite tracks (animal movements) with physical environmental data such as sea ice
• Involvement in marine bioregionalisation exercises, in the Southern and global oceans

Past data mining projects

• Observations of seabirds at sea
• State of the Environment Reporting
• Gas chromatograph analysis
• Leopard seals on Macquarie Island
• Computer-assisted discovery of scientific knowledge

Publications

Request a copy of any of these publications.

Journal and conference publications

Draft or in-press publications

1. Schwarz, J.N., Raymond, B., Williams, G., Marsland, S., Pasquer, B., Mongin, M., and Wright, S. (2009) Climatological anomalies in wind, sea surface temperature, sea-ice and chlorophyll concentrations during the BROKE-West survey. Deep-Sea Research II, in press.
2. Woehler, E.J., Raymond, B., Boyle, A., and Stafford, A. (2009) Seabird assemblages observed during the BROKE-West survey of the Antarctic coastline (30°E—80°E), January – March 2006. Deep-Sea Research II, in press.

Software

The Polar Toolbox is a collection of Matlab functions for various analyses of Antarctic data, but which might be useful for analyses of other data. Most of this code has been tested under Matlab 2008a. It may also be compatible with other applications, such as Octave, which is a freely-available, mostly-Matlab-compatible package.

All code here is experimental and made freely available. Before downloading you will need to register (free!) with the Australian Antarctic Data Centre. This allows us to direct our development attention to the most active files.

Download the toolbox

Queries?

Toolbox contents

% Polar Toolbox: A Matlab toolbox for various Antarctic analytical tasks
% Ben Raymond ben dot raymond at aad dot gov dot au
% September 2009
%
% aloc - Belbin's ALOC non-hierarchical clustering algorithm
% angle_normalise - normalises angles to the range -pi to pi
% bezierfit - cubic bezier curve fitting, given four control points
% cellmax - the maxima of a cell vector of identically-sized matrices
% cellmean - the mean of a cell vector of identically-sized matrices
% cellstd - the SD of a cell vector of identically-sized matrices
% cluster_clara - clustering using R's CLARA function. Requires R and the matlabRlink toolbox to be installed
% cutree - cuts a dendrogram and return the group membership vector 
% dump_csv - write numeric matrix to comma separated text file
% gebco_colormap - colormap similar to that used on GEBCO bathymetric charts
% gb - builds network structure from data matrix, following Raymond & Hosie 2009
% get_ice - retrieves ice data from binary files as obtained from NSIDC/seaice.de/etc
% get_icemotion - retrieves ice motion data from binary files as obtained from NSIDC
% get_sst - retrieves sea surface temperature data from binary files as obtained from NOAA etc
% icemotion_easegrid_ll2rc - converts longitude/latitude coordinates to row/column coordinates in the ice motion EASE grid
% indicator_spp - calculates indicator species from clusters, following Dufrene & Legendre (1997)
% join - join a cell array of strings or numeric vector into single string
% mean_nonan - calculates means, ignoring NaN elements
% ndvi_colormap - colormap similar to that used to show normalised vegatation index images
% proplim - confidence interval of a proportion m/N
% ranks - ranks of x adjusted for ties
% slidingmean - applies a sliding mean to a vector
% slidingmedian - applies a sliding median to a vector
% spearman - computes Spearman's rank-order correlation coefficient for two vectors x and y
% split - split a string into substrings
% std_nonan - calculates standard deviations, ignoring NaN elements
% streampatch - render 2-d streamlines using patch arrow objects
% sunriseset - calculates sunrise and sunset times
% wiifigure - allows a Matlab figure to be panned, zoomed, and rotated by a Wii remote device