Appendix 2 - Data entry, error checking, and loading
The data in nonfish_bycatch have come from the fishing industry.
This section outlines the flow of paper-recorded data, from
collection through to its availability to researchers for analysis,
and defines the separate tasks that are required to do this.
In summary, the nonfish_bycatch data are recorded on hand written
paper forms. Each trip is identified by a unique trip_key, each tow
or set by a unique station_key, and each capture by a unique
1. Pre-key entry, visual checking and batching:
The data are forwarded via Mfish, to a project
team member, who checks the forms, and forwards the data to key
2. Key entry of data:
At this point, trained data entry operators key in the data from
the collated forms to an electronic fixed format ASCII file format on
computer by keyboard entry. NIWA uses the KEYS Data Emulator for data
All data entry is verified, that is, each page of data is keyed in
twice and the two results are cross-checked for mismatches. Any data
entry operator errors are corrected at this point.
The electronic data files are transferred for error checking along
with the original raw data file. At this point the data are now ready
for error checking and formatting routines.
3. Data error checking, validation, and grooming:
Data files are put through a number of computer error checking
(validation) routines that look for inaccuracies and inconsistencies
within trips. Any errors detected are corrected. Data are then passed
through these errorchecking routines until the data reach a
satisfactory standard that will allow them to be inserted in the
appropriate database tables.
The data are inserted into "working tables". This allows
further checks of the integrity of the data, by taking advantage of
relational databases ability to manipulate, match and compare related
sets of data.
4. "Groomed", validated data loaded to
Available for analysis: The clean, groomed, and validated data are
inserted into the appropriate database (in this case nonfish_bycatch)
and now become available for extraction and analysis.
The clean electronic data files and raw paper data are then
archived for safekeeping.