ABE's data structures contain mostly raw data straight from ABE's logger. When I have massaged any data item, that will be noted.
These data structures reflect some artifacts of ABE's architecture. ABE's data logging is tied to a 10 second cycle, which coincides with the long baseline navigation (LBL) cycle, the imagenix scans, and the electronic photographs. We also record some key dynamic data on a one second cycle.
All data items are time-stamped. To my knowledge, these times are accurate to better than 1/10 of a second, which is the sampling rate of ABE's highest bandwidth control processes.
If you don't understand Matlab data structures, you'll need some basic fluency with them. They are used in a very simplistic way, however.
At the end of this document, a set of data files will be defined.
Units and Time
Unless otherwise noted, all angles will be in degrees, all distances in meters, and all times in seconds. Time is represented as seconds in the day. If the day rolls over (it usually does in the course of an ABE dive), time does not reset but keeps on going.
Angles observe the following conventions:
ABE's data logging cycle
Every 10 seconds, ABE's top level controller (#ABE) sends the navigation computer the latest returns from the LBL receiver. It then queries the navigation computer for a number of blocks of data. All these transactions are captured in ABE's data logger. The relevant blocks are then parsed out and loaded into Matlab data structures.
The raw acoustic travel times are recorded in a Matlab data structure called ttx. this structure has the following fields:
ttx =
t: [2264x1 double]
tt: [2264x4 double]
In this example, there were 2264 samples taken. The t field represents time (in seconds), the tt array contains the raw 2-way travel times in milliseconds (sorry, an exception to the rule about representation of time).
After #ABE sends the travel time info to the navigation computer, it then queries the navigation computer for a series of blocks. These are listed below and are updated every 10 seconds
c1 =
t: [2264x1 double] % time in seconds
hdg_d:
[2264x1 double] % desired heading
hdg: [2264x1 double] % composite heading used by ABE
in real time
z_d: [2264x1 double] % desired depth (positive down)
z: [2264x1 double] % depth used by ABE in real time (see
C3 for raw)
xfix: [2264x1 double] % x position of raw fix
yfix: [2264x1 double] % y position
x: [2264x1 double] % estimated position used by ABE for
track following
y: [2264x1 double] %
altitude: [2264x1 double]
% filtered altitude, used by ABE's bottom following
xprime:
[2264x1 double] % distance off the desired trackline
eol: [2264x1 double] % distance to end of line
recorded every 10 seconds
c2 =
t: [2264x1 double]
% time
thrust: [2264x7 double]
% commanded thrust to each of 7 motors (N)
speed: [2264x7 double]
% rotational speed of thruster (units??)
recorded every 10 seconds
c3 =
t: [2263x1 double]
flux_hdg: [2263x1 double] % flux gate compass heading, unwrapped
gyro_hdg: [2263x1 double] % raw gyro heading, unwrapped
gyro_drift:
[2263x1 double] % gyro drift estimate
gyro_error_rate: [2263x1 double]
% estimated rate at which gyro drifts
gyro_error_offset: [2263x1 double] % offset
term in linear drift model
gyro_error:
[2263x1 double] % diff between corrected gyro and flux gate
depth_raw: [2263x1 double] % raw measured depth
altitude_raw:
[2263x1 double] % raw measured altitude
pitch: [2263x1 double] % raw measured pitch
roll: [2263x1 double] % raw measured roll
Note that c1,c2,c3 are not necessarily the same length (they might be). You should keep each data item with the proper time vector in any case.
ABE also records attitude, depth, and altimeter data at one second rates. Every 10 seconds, #ABE gets the records for the 10 previous seconds. The time stamps are correct. These values are greatly superior to the ones in the C1-C3 structures, as they are sampled more often and are not filtered in any way.These are contained in the following:
att =
t: [13103x1 double]
flux: [13103x1 double] % unwrapped flux gate
pitch: [13103x1 double] % pitch angle
roll: [13103x1 double] % roll angle
z: [13103x1 double] % depth, unfiltered
altitude:
[13103x1 double] % altitude, unfiltered
gyro: [13103x1 double] % unwrapped gyro
gyro_correction: [13103x1 double]
% gyro correction
hdg: [13103x1 double] % post-dive heading from both flux
and gyro
By "unwrapped" for heading data, I mean that the values are not modulo 360. For example, if ABE started pointing due north, then spun around 10 times to the right, it's unwrapped heading would be 3600. These are convenient, since you can run filters on them directly.
The att.hdg field is a conventional heading (0-360). It is the only element of this structure that was massaged after the run and was produced in several steps. First, the drift error is removed from the gyro by beating it against the flux gate over a period of 10s of minutes. Then, the wobbles in the flux gate are removed by comparing it to the corrected gyro using a sinusoidal model. Then, the gyro is adjusted again based on the corrected flux gate.
Every 10 seconds, ABE completes a 7 second scan with the imagenix. The raw data is contained in the ima structure. When no returns are received, a value of -.35 meters appears in ima.range
ima =
t: [35650x1 double]
angle: [35650x1 double]
range: [35650x1 double]
The reprocessed navigation is contained in a structure called pns. These items should be used instead of the real-time estimates, as more fixes can be recovered and more ranges used off-line than in real time. While bad acoustic returns are weeded out, these results are not smoothed in any way. The pns structure contains the following fields:
pns =
t: [1751x1 double]
x: [1751x1 double]
y: [1751x1 double]
z: [1751x1 double]
r: [1751x4 double]
error: [1751x1 double]
num_good: [1751x1 double]
xfiltered: [1834x1 double]
yfiltered: [1834x1 double]
zfiltered: [1834x1 double]
tfiltered: [1834x1 double]
Note that the filtered and unfiltered data are of different lengths.
This happens because the filtered
data is first intepolated onto ABE's 10 second cycle, then the filter
is applied.
ABE's science sensor data, along with some vehicle status information, is contained in a structure called sci. It has the following fields:
sci =
t:
[2264x1 double]
maggy: [2264x3
double] % maggy voltages, 24000 times the output voltage
total_field: [2264x1 double] % uncalibrated
estimate of total field
obs: [2264x1 double] % optical backscatter output in volts
volts: [2264x4
double] % voltage on ABE's battery packs
charge: [2264x4
double] % total charge removed from ABE's batteries
charge_rate: [2264x4 double] % charge
rate (current) from batteries
watts_pack: [2264x4 double] % watts
from each battery string
watts: [2264x1
double] % total watts
ftemp: [2264x1
double] % frequency output from Seabird temperature sensor
fcond: [2264x1
double] % frequency output from Seabird conductivity sensor
temperature: [2264x1 double] % Seabird
temperature using their formula (note,
cal constant are out of date, unit has been recalib'd)
The sound velocity profile used by ABE to reprocess the navigation is contained in:
svp =
depth: [93x1 double]
velocity: [93x1 double]
The transponders used to navigate ABE are contained in:
xp =
name: 'ABCD'
freq: [8.5000 9.5000
11 13.5000]
position: [4x3 double]
% one row for each xpndr, columns are x,y,z
n:
4
color: 'rgbk'
Background contour lines were created from seabeam data. These were used for real-time display for ABE's surface nav, and for plotting afterwards.
b =
1x212 struct array with fields:
depth
x
y
index
This is a Matlab structure array (as opposed to the previously defined arrays of structures. If you understand the difference, you'll have the matlab structure thing down cold). Each element of the array contains a scalar (depth), two vectors (x,y) whose length can be different for each array element, and an index, which can be used to plot by color.
A matlab function written by me called plot_background(b) can be used to display this information. Plot_background does not destroy the current plot, so it can be called after tracklines have been plotted.
The dddate structure contains the date, including the month, day, and year. Note: not Y2K compliant! You will have to remember whether ABE ran in 1899, 1999, or 2099.
ddate =
day: 24
month: 2
year: 99
The org data structure contains the origin for the xy coordinate system
used on that dive. This information allows you to convert any xy coordinates
to lat/lon. You can use the attached utilities
to perform the conversions.
org =
lat: -17.4180
lon: -113.2142
lon_deg: -113
lon_min: -12.8530
lat_deg: -17
lat_min: -25.0800
The tl structure contains the end points of the tracklines
tl =
x: [58x1 double]
y: [58x1 double]
The rng structure contains the range measurements from the vehicle to
the transponders. These are slant ranges, not projected xy ranges. The
rraw
field contains all travel times mapped to ranges, the rclean fields
contains the ranges that remain after flyers have been removed.
rng =
t: [2845x1 double]
r: [2845x4 double]
rraw: [2845x4 double]
rclean: [2845x4 double]
The stats structure contains some useful information about the dive. The ddate structure is included as well.
stats =
dive_number: [17x1 double]
tstart: [17x1
double]
tbottom: [17x1
double]
tend:
[17x1 double]
distance: [17x1 double]
num_pix: [17x1
double]
ddate: [
1x17 struct]
The processed scanning sonar data structures.
The processed nav data, attitude data, and raw sonar and the parameters used to create the dotcloud are contained in two data structures. While these can be named anything, I refer to them as s and p. They are created by process_dotcloud and are handy for plotting and rerunning the dotcloud creation process with different parameter. p contains the sensor offset data and a few other parameters that control the transformation process (i.e. how the imagenix was pointed, whether to use the gyro or the fluxgate, etc.) The s structure contains the processed nav (from process_edl), attitude data, and raw imagenix data (with short returns clipped by process_dotcloud)
p =
roll_ducer_protractor:
2.2000
pitch_ducer_protractor: -1.3000
heading_ducer_protractor: 1
roll_body_protractor:
1.5000
pitch_body_protractor:
0.1000
heading_body_protractor: 0
roll_body_computer:
-0.7000
pitch_body_computer:
0.0050
heading_body_computer:
0
roll_body_offset: 2.2000
pitch_body_offset: 0.0950
heading_body_offset:
0
roll_ducer_frig_factor: -2.1000
pitch_ducer_frig_factor: 0
heading_ducer_frig_factor: 0
dive_name: 'J230_T63'
mes_orient: 'forward'
heading_source: 'fluxgate'
s =
t: [31431x1 double] % time in seconds since unix was born
dotcloud: [31431x3 double] % x,y,z of dotcloud
vehicle: [31431x6 double] % x,y,z,roll,pitch,hdg (angles in
radians)
% the heading is from the gyro
ducer: [31431x6 double] % x,y,z,roll,pitch,hdg of sensor relative
% to vehicle cg
fluxgate: [31431x1 double] % fluxgate compass heading
angle: [31431x1 double] % bearing of imagenix head
range: [31431x1 double] % range measurement from imagenix
scan_number: [31431x1 double] % increments by one each time
angle
% changes direction
transect: [31431x1 double] % transect number
roll_ducer_offset: -1.4000
% total angle offset computed from
% all the offsets in p
pitch_ducer_offset: -1.4000
heading_ducer_offset: 1
For ABE, nearly all the offset parameters are zero, as the sensors and sonar are zeroed to the best of our knowledge. Of course, there are offsets, and we determine these from the data and enter them into the "frig_factor" entries.
Operations on dotcloud structures
First, you can rerun the dotcloud computations with altered parameters (in the p structure, for example) by running
snew = rerun_dotcloud(s,p,nskip)
where nskip is an optional parameter that lets you skip every nskip points.
this makes a new s structure from the s and p provided. For example, if you change p.roll_ducer_frig_factor and you do rerun_dotcloud, you get a new dotcloud based on that new set of parameters.
you can copy a structure simply using matlab
s2 = s;
you can concatenate two structures
s12 = scat(s1,s2)
you can extract elements from s
strimmed = sextract(s,ind)
where ind is an array of indices for the points you want. Example, to get all the dotcloud points above a certain depth:
ind = find(s.dotcloud(:,3) > MAX_DEPTH);
sclean = sextract(s,ind);
Simple plots on scanning sonar data
A routine can be used to view the data. These routines do not grid the data, they plot the data as colored points.
view_bathy(s,p)
then type rotate3d and have fun. You can also step through the individual scans with:
view_scans(s,p,1)
where the last parameter determines whether to pause between scans or not.
Data Files
For each dive, two data files will be produced. The first, called abenn.mat (nn is the dive number) will contain the following structures:
att (one second attitude/depth/altitude data)
pns (post-processed navigation data)
sci (science instruments, ABE power usage)
ima (raw imagenix data)
xp (transponder locations)
svp (sound velocity profile)
background (seabeam bathymetry)
c1, c2, c3 (vehicle realtime data)
ddate (date)
stats (some useful info about the dive)
rng (raw and processed LBL range data)
org (origin used for lat/lon to xy conversion)
Also, the processed scanning sonar data will be contained in a file
called abenn_dot.mat. It will contain the s and p structure for the dive.
Some useful Matlab files (xy-ll conversion, ls nav solution)