Appendix 1 - Data entry, error checking, and loading
The data in regional_ce have come from companies with New
Zealand vessels fishing on the high seas and furnishing non-NZ Catch
Effort forms. Data from CCAMLR trips are supplied in electronic form.
These data from CCAMLR trips are not subject to the same level of
checking by NIWA, as would be expected if NIWA was supplied with the
raw data and was responsible for the data entry of these data. That
is the data from CCAMLR trips do not pass through the data entry
stages described here.
This section outlines the flow of paper-recorded data, namely for
the South Pacific Regional Purseseine
fishery, and defines the separate tasks that are required to do
The South Pacific Regional Purse-seine data are supplied on forms
as either hand written paper or typed format. Each trip is assigned a
unique trip number and each set a sequential station number.
The date and time will also be recorded as part of the station
1. Pre-key entry, visual checking and batching:
The data are then forwarded via the Ministry
of Fisheries, to a project team member, who checks the above, and
forwards the data to key entry.
2. Key entry of data:
At this point, trained data entry operators key in the data from
the collated forms to a electronic fixed format ASCII file format on
computer by keyboard entry. NIWA uses the KEYS Data Emulator for data
All data entry is verified, that is, each page of data are keyed
in twice and the two results are crosschecked for mismatches. Any
data entry operator errors are corrected at this point.
The electronic data files are transferred for error checking along
with the original raw data file. At this point the data are now ready
for error checking and formatting routines.
3. Data error checking, validation, and grooming:
Data files are put through a number of
computer error checking (validation) routines that look for
inaccuracies and inconsistencies within trips. Any errors detected
are corrected. Data are then passed through these error-checking
routines until the data reach a satisfactory standard that will allow
them to be inserted in the appropriate database tables.
The data are inserted into "working tables". This allows
further checks of the integrity of the data, by taking advantage of
relational databases ability to manipulate, match and compare related
sets of data.
4. "Groomed", validated data loaded to
database. Available for analysis:
The clean, groomed, and validated data are
inserted into the appropriate database (in this case regional_ce)
and now become available for extraction and analysis.
The clean electronic data files and raw paper data are then
archived for safekeeping.