Import Data Files

Manual loading of data is handled in Import Data Files under Data Setup. CSV files and TOA5 DAT files (Campbell Scientific variant of CSV) are the two formats accepted.

CSV is a simple universal format, but it hides significant complexity. This service guides you through properly identifying all the details necessary to import data properly.

Caution

When you load a new data file, you will be creating a new table in the database. It is permanent. Do not load data without double checking to make sure this is the correct file.

Timestamps and Time Issues

Caution

Confirm what time the CSV is in. Was the logger set to daylight savings?

The Dendra team recommends always using standard time for the time zone and never applying daylight savings. Time will be converted to UTC (Greenwich Mean Time) internally, with the time zone preserved.

How to Import a CSV File

I. Initial Configuration

1. If you have manual measurements (i.e. readings taken without a datalogger, select Organization Import. Otherwise, select Import for a Specific Station.

2. Select a station. You must create a station within Dendra before you can import data from that station.

3. If this is the first time you have loaded data from this table: Choose New File Upload and Import.

   If the table already exists and you are loading additional data: Select it from the list that shows up in the dropdown menu.

4. Hit Continue.

II. Upload

1. Select a destination. Again, here you have the option of importing into an existing table. Otherwise, type in the permanent name of this table for the station.

   Tip

   Naming Conventions

   • The table must be in snake case (lowercase with underscores between words ( e.g. table_snake_case)).
   • The name can be a duplicate of other table names elsewhere, such as tenmin for ten minute data.
   • The tables have their own name space under the station. No need to add station name to the table.

   Comment: This should start with a version of the table name. It has no restrictions - it is for context. The comment will show up when you try to import more data as a selectable option in the interface.

2. Specify processing options. There are only two choices.

   Option
   Generic CSV	Thirteen options are given for processing the CSV. Each option provides some instruction on how to use. Most logger output can be handled with the proper application of these options.
   TOA5 (Campbell Scientific)	This is a .dat file. The DAT files are a specific form of CSV and selecting it will automatically set the options appropriate for .dat file format.

3. Upload file and validate. Choose a CSV file to upload. Once you select the file, the validation system will kick in. The interface will change.

   You will see two new options: Dry Run and Live Import.

   The system is trying to transfer the file, so if it is large, please be patient. By default, this is a Dry Run. The file is being uploaded to a workspace for you. It is not being ingested into the database.

4. If it succeeds, the interface will change again. You will see a tabular form of the CSV at the bottom of the screen. Please check to see if this looks correct.

   Note

   Troubleshooting

   If the file does not load, this means your settings were not right. Please go over the options. When in doubt, check the Stryp Byte Order Mark (BOM) box. That may solve the issue.

   If you see data and it looks correct, Select Live Import.

III. Live Import

This is going to create a new permanent table in the system.

Caution

Dendra does not delete tables.

Perform this carefully. The interface has several checks designed to make you think before uploading.

1. Confirm The TIME in the preview data is correct and represents my observations in UTC using the checkbox.

   Note

   The timestamp needs to be converted to UTC. If you have not adjusted the data in the options, please go back and do so. All data must be converted to UTC for central storage.

   However, you must tell the uploader how many seconds the shift is. Also, if there is a Daylight Savings change mid-file, you will need to either split the file in two or change the timestamp to be timezone aware.

2. Scroll back to the top of the page. On the top right, there will be a blue New button with the name of the table name you gave in snake_case on the previous screen. There is a Copy icon to the right of the name. Click it.

3. Scroll back down. Click into the text. Paste or type the table name exactly as it appears above. This is a security measure meant to ensure you know what table you want, and what time you are setting for the table.

   If you copied correctly, Begin Live Import will now be active.

4. Click Begin Live Import.

You may not get confirmation that the import is completed. The process will continue whether or not you stay on the page.

At this time return to Stations and check your Station Readiness Checklist.