Comments, queries or suggestions can be emailed
To Programmer's Report
Data Quality Report
Ruth Lawless, October 2001
Data Currency Voyage Description Logging System Data Sampling Interval
Data Gaps Software Data Quality
Data Start Time: 25 September 1991 05:20 UTC
Underway Data End Time: 26 October 1991 18:50 UTC
This was a marine science voyage. Technical support personnel were provided on board, but underway data were quality checked well post-voyage (October 2001).
|29 - 30 September||Macquarie Island|
|03 - 07 October||Hobart, repairs for marine science|
|08 - 26 October||Marine Science|
DLS data types were logged.
4. Data Sampling Interval: 60 seconds
5. Data Gaps
Underway data were quality checked using Java based software created by Gordon Keith and Peter Wiley. Depth data were already processed, but it is not documented when, how and by whom this was done.
Back to Contents
7. Data Quality
of Checked Parameters
The following is a description of what criteria were used to flag data as "good" or "bad". Underway data users should also refer to the section on Data Gaps.
Bathymetry (Water Depth)
There is no documentation relating to the processing protocol or quality of these data.
Ship’s Heading (SH)
SH is a measurement of the direction that the bow of the ship is facing.
These data were accepted as good.
Ship’s Speed (SS)
Ship's Speed data are derived from the electromagnetic current meter output (Ship's Log). SS is a measurement of ship's speed through the water and is directly affected by waves.
Ship's Speed data are believable with respect to the ship's schedule and capability and have therefore been left as good.
Fluorometer Flow (FLU_FLOW) and Fluorometer
FLU_FLOW are all believable and were left as good.
FLU_VALUE data were rejected were flow is low or erratic. A section of the trace on 04 October doesn't resemble real data and was also marked bad.
There is no record of when or if the fluorometer was cleaned and calibrated during this voyage.
Thermosalinograph Parameters: Salinity (TSG_SALIN),
Conductivity (TSG_CNDUCT) and Temperature (TSG_TEMP)
Accurate TSG_SALIN data rely on flow through the uncontaminated sea water (USW) line. Therefore, salinity values were marked bad where FLU_FLOW is low or erratic. Extreme outliers were also rejected. The remainder of TSG_SALIN data were left as good.
TSG_CNDUCT data points were allocated good or bad quality flags to match those of corresponding TSG_SALIN data points.
TSG_TEMP data are not an accurate measure of sea water temperature and were rejected as a matter of routine.
Water Temperature - low resolution (W_TEMP)
The accuracy of these data is inherently suspect due to the position of the temperature probe. The sensor was located about a metre within the uncontaminated seawater intake relatively high on the hull and was subject to freezing when in pack ice and taking in air during heavy pitching.
W_TEMP data were rejected where USW flow (i.e. FLU_FLOW) is low or erratic. Sections of particularly variable data were also marked bad. Although the remainder of W_TEMP data were left as good, users should acknowledge their limitations.
Air Temperature Port and Starboard (A_TEMP_P
Warm air from the exhaust funnel aft of the air temperature sensors corrupts air temperature data when upwind of them. As apparent wind direction data for this voyage are not loaded onto the marine database, it could not be determined when this occurred.
Therefore, data from both air temperature sensors were marked bad where they differ by more than about 1.0º C or where the shapes of the traces don't correlate well. Outliers were also rejected.
Barometric Pressure (BAR)
Apart from one outlier (value = 0), BAR data were accepted as good.
Humidity Port and Starboard (HUMID_P and
The humidity sensors are located close to the air temperature sensors and humidity data are therefore also affected by exhaust air from the funnel, although to a lesser extent. When a sensor is downwind of the funnel, humidity data are sometimes decreased relative to the unaffected sensor. However, as with air temperature data, humidity data could not be quality checked with respect to apparent wind direction.
Data from both humidity sensors were marked bad where values differ by more than about 10% or where the shapes of the traces don't correlate well. Outliers were also rejected. Starboard output was faulty between 23 October 12:54 UTC and 25 October 02:37 UTC. Data from the port sensor were left as good during this period.
Photosynthetically Active Radiation (LICOR_R)
Note that this sensor is affected at times by shadows from the ship's superstructure.
Apart from extreme outliers, these data were left as good.
Wind Direction True Port (WDPT) and Wind
Speed True (WSPT)
Apart from a few outliers, WDPT and WSPT data were accepted as good.
Apparent wind direction and wind speed data should exist, but have not as yet been loaded onto the marine database.
Back to Contents
John Chrisoulakis and Dave Watts, 15 October 1991
1.0 TASKS ACHIEVED
1.1 Hardware installations prior to departure
A DEC 386 PC and a Macintosh Si were installed in the
conference room along with 2 HP Laserjet III laserprinters.
The PC and Mac were connected via thinwire Ethernet using
PCSA and Pathworks for Macintosh respectively. The PC could
not use the network because the version of PCSA was too old.
1.1 Software installations prior to departure
VMS 5.4-2 was installed on both Aurora and Tiny by Mark Conde and
Howard Burton. Pathworks for Macintosh V1.0 was installed on
Aurora. An Appleshare disc was created on Aurora called 'VAX
1.2 Post-processing software
All shipboard personnel had access to the post-processing
system "LOOKSEA". Some enhancements and bug fixes were carried out
CHART was again never used for post-processing of logged data.
The birds at sea database was updated continually during the
The following is a list of improvements and major fixes done
during the voyage.
- added oxygen and oxygen temperature to CTD plotting and listing
- added more documents to the documentation menu. Notably a general
guide on the comupter facilities, a user guide to running the
telex facility and a guide to booting up TRAC.
- created a menu for maintaining voyage reports.
- fixed the CTD pre-processing program to understand the data
format of the CSIRO CTDs and extended the bottle data files
from 12 to 24 bottles. Improved the error message from this
- moved several programs that resided outside the LOOKSEA area
into appropriate directories. Greater care must be taken in
ensuring the LOOKSEA application is not split.
- fixed bugs in the program that checks the qualtity of the
DLS tapes. This was run for every tape and all passed.
- created a looksea_dev:[log] directory so that all log files
were written to one spot and not scattered throughout the
- added a field to Powerhouse screens for seawater flow. Bug fixes
to the data averaging reports to allow for any year.
- tidied up the CTD programs for creating transects, T-S plots
and downcasts and made them use a more sophistiacted labelling
- rationalised the 4 versions of the NCAR translator into 1
- major tidy of Tiny's system area and moved Powerhouse onto its
system disc to balance the disc usage.
- added a plot every Nth point option to the DLS property versus
- development of DLS_EXPORT for extracting DLS data into ascii form.
- cross calibrated the sea water line temperature and salinity
with CTD measurements. This confirmed the linearity over ten
degrees and one ppt respectively.
1.3 TELEX facility
The Telex facility was available with a limit of 200 words for
personal telexs. No changes made.
1.4 VMS System work
Aurora's system area was tidied up so that the system command
files more closely reflected what happens at the Division.
- created the OPER menu and added the command procedures for
bulk user create and delete.
- set default values of working set etc to more realistic values
and increased the page file to 50000 blocks from a miserly 17000
2.0 PROBLEMS ENCOUNTERED
2.1.1 HP and Scientific instruments
A repeat problem for DLS HP-7979 tape drive number two as it
displayed on at least two occasions an incorrect number of dots in
its "tape used" bargraph indicator. At these times the display
should have contained seven or eight dots. However, only three
dots actually were displayed.
Met program running on HP Vectra in Met
office only shows
correct relative wind speed and direction when the ship was
stopped. DLS seems to record the correct data and values agree
with the Steedmans Met station. Reboot of HP Vectra only gives
about two good readings before it gets ships speed log and then
produces incorrect wind data.
The instrument room Magnavox satellite navigator got into a state
such that it alternated between two positions. It was DR but DLS
said nav ok - so screens showed our position alternating between
the two - restart of MX made position come good. This problem
happened twice. The second time it recovered on its own.
DLS was logging GPS unavailable and no satellites whilst data
summary screen was showing NAV OK. GPS at this time had four
satellites. Reinitialisation of DLS navigation caused it to pick
up GPS again. This problem occured once. Otherwise GPS was good.
Humidity values sometimes logged at one tenth what they should
be. eg 8 instead of 88; The starboard humdity sensor died in the
last week of the voyage.
When the EK500 is off, the data summary screen shows biomass
of 33312 although 0.0 is logged. The biomass value displayed by
the DLS data summary screen was the last value logged when the
EK500 ceased transmitting data.
The HP laserjet IIIP printers gave problems. The printer with
Postscript cartridge worked most of the time but was flakey on
big jobs (ie about 10,000 blocks). The other laser was set up
as an HPGL plotter/text printer. However we have not been able to
get the HPGL printer to print an HPGL plot. This was not regarded
as important since all text documents can be sent to the other laser
and no software produces exclusive HPGL plot files.
Also the WPS PLUS printouts sort of worked.
The bridge console received data but was otherwise dead. It would
not accept any keyboard input such as fixes. It appears that after
the most recent DLS crash, when the system was rebooted, the
bridge console then started operating correctly again.
The 20 Mbyte disc drive on the CTD PC has proven to be rather
awkward. With the deeper CTD dips going to over 4000 m the disc
can only hold one or two dips before they have to be deleted. A
check for adequate disc space has to be done before every dip.
XBT's caused some problems if the launch was two minutes after the
load. After loading an XBT and waiting, the PC thought it had
failed. It was then launched and looked fine. Extraction from the
logging files showed two files per launch. The first contained
the correct header and only three or so lines of data. The second
file contained the correct depth-temperature data but the header
had a ship name of XXXX and the lats and longs were swapped over.
Not sure if this is a problem for XBT_SLURP or DLS or the XBT PC.
In any case there was an easy work-around by editing the data
Decserver Lewey did an auto-logoff twice. Also one line to a
terminal from Lewey Port 4 was flaky. It just as mysteriously
2.2.1 T.R.A.C. V
TRAC died 4 times.
1) Crash at midnight on V1. Found dead next morning with no obvious
symptoms. Cold restart required.
2) Crash two days later. Possible DJW finger trouble with tape
drives. A warm reboot worked.
3) Only one user initiated action which hung the system. Whilst
changing the database and selecting a line for an ETA point,
got 'Pascal error 5 - TRAC error 395'. It would continually
loop with this error. A warm boot did not clear it. Did a
cold boot. Checked the database and found that a point had
been entered twice. This point was on the line that was
selected. Cleaned out the duplicate point and removed any
4) The dreaded TRAC error 59 occurred once. A few minutes after
doing some extensive manipulations of the TRAC database. Used
"SHIFT-RESET RUN" to restart the system.
2.2.2 CHART V
It is possible to lose data after a LAN failure. Under some
conditions the LAN file is overwritten instead of appended to.
The "Close, rename, and reopen" option of Chart's lan server
manager was observed to fail. Detailed descriptions of these
events are found in the "Computer Log" book for the WOCE91
The VMS documentation set is ancient - all version V5.0. The
Fortrans manuals are even more so being for V4. One of the C
manuals is also pre version 5. Qubit must supply what we pay
Still no manual for Chart.
The Mac can see and use the network resources (file and print)
but the installation of MacTerminal failed. Copying the
instructions lead the DEC-supplied installer expecting the
folder of system resources to be on a disc called "Installation
Folder" and not search the top directory of the hard disk. Trying
to beat the system by placing these resources on a floppy with the
name "Installation Folder" lead it to say that the contents of the
floppy were not what the installer script expected.
The laserprinter with the postscript cartridge was initially set up
via a serial line from the VAX at 9600 baud. This worked fine for
all VAX print jobs but the Mac failed to print to it even though
the chooser could see the printer. It is possible the error lies
with the VAX print symbiont. This was not fully pursued as the
laserprinter could be more efficiently used by connecting via the
Appletalk line at 232,000 baud. With large plot files from the VAX,
the difference was obvious. An example was a 10,000 block plot file
at 19200 baud took one hour via the serial line and only half an
hour via Appletalk.
The version of the latest VAX laserprep file is V6.0.0 which is
not compatiable with the latest version for the Mac (V6.1). All
Macs and Vaxes had to have version 5.2 (the latest common version).
3.1 Urgent work
The met PC problem needs to be corrected for future voyages.
3.2 PC/Macintosh facilities and word processing
3.3 Data quality assurance and cleaning
Data logged needs to be examined and bad data flagged in some way.
Using mouse driven graphics to mark data with the correct quality
flag is a possible solution.
3.4 Hardware enhancements
On this cruise there was zilch demand for the HP Draftmaster
plotter. All plots were sent to the lasers. No enhancements are
required except to finish the appletalk connection to the lasers.
Currently it runs across the floor of the conference room.
Get another Postscript cartridge or ditch the printers. Running
in postscript mode is infinitely more flexible. After all what
software produces HPGL that can't be done in Postscript. The
CSIRO Sun workstation had a postscript viewer and all print jobs
were transferred in Postscript to DOS disks and then into the Mac
via Apple File Exchange to be printed by SendPS. Simple !!!
The laserprinters were very slow.
3.5 Software enhancements
3.5.1 System changes
The following software upgrades should be carried ASAP.
- upgrade AURORA to the latest version of PCSA and put
windows on the PC before it drives us mad.
3.5.2 Future improvements to DLS
DLS needs to get some EA200 settings. In particular, if possible,
some form of quality flag needs to be recorded. Also the
"minimum depth" setting is not recorded. There is
no way of telling from the logged data if these are real
features or if they have been clipped at the top due to the
minimum depth being greater than the actual depth.
3.5.3 Future improvements to LOOKSEA
Merge XBT_SLURP into ALF_BUILDER so that all XBT data are
Currently the structure of LOOKSEA has been written in such a
manner that it is easy to work on data from a particular cruise
but not to simultaneously work on data from multiple cruises.
To get the best out of it will require fundamental changes to how
LOOKSEA operates. Further WOCE cruises will require cross
comparisions of data from prior WOCE cruises in order to make the
best decision about when to do any CTDs etc. It is also of
importance to the Division with it's repeated cruises to Pyrdz Bay.
To help this cross-comparision of data, a database of stations done
for all cruises should be constructed and seeded with data from
all CTD's etc from past cruises. Currently LOOKSEA uses a text file
to describe station activities but a Powerhouse application may
serve users best. Simple questions such as what happened within
so many miles of this spot or where are all the current meter
strings deployed can then be addressed.
Some work was done 4 years ago by DJW in comparing data from CTD
bottles with the temperature and salinity profiles. Division
staff should be asked if such a facility be made so that all
bottle data can be recorded via LOOKSEA instead of on individual PCs.
In the past, no attempt has been made in properly calibrating the
salinity recorded from the CTD by measuring the salinity from the
bottles. To fully use the CTD data, it would seem sensible to
measure salinities from bottles on a regular basis.
3.6 The question of "clustering" AURORA and TINY - Mk III
We list here advantages and disadvantages which we see in clustering
the two machines. Tiny is the router for the network and it is
not clear what effects on cluster/failure this could have.
. one system disk
. more disk space free on TINY
. easy sharing of disk resources
. one set of user accounts and one UAF. Users would not get so
confused about what went where.
. cluster-wide access to printer resources. With the Vaxes as
separate nodes The Laserwriters on Appletalk can only be accessed
via Aurora unless one reinvokes the Lasersquirt technique.
. single-backup strategy
. requires more experienced system managers but tends to be trivial
. additional licensing costs. Would require VaxCluster.
Effects of Aurora dying
This would be critical wether in a cluster or not. As for Tiny,
the only applications that would be affected are the Birds
at sea, Fish, Whales and Seals and any WPS users. The later can
migrate to WYSIWYG WP on the Mac or PC. This has the advantage
of crew and passenger access to word processing when the computer
system is not on.
Effects of Tiny dying
Any printers attached to the Mac via appletalk could not be
used by any of the Vaxes if Tiny was dead. Not a problem as
you could invoke a queue on Aurora via a server. One laser could
be connected via Appletalk and the other serially so that both
the Mac and the Vax could access a laser printer.
Interestingly Powerhouse would proably run on Aurora but only
give a warning that it's license was invalid.
Finally, we both think that one programmer would be sufficient
for future voyages. DLS was stable. Most of the data analysis
programs have been developed and provided no major changes were
planned, one person should be able to maintain the system.
Back to Data Quality Report