Data logger is here!

After months of planning and designing, I’m excited to announce that the all-in-one SDI-12 data logger is here! I got the board printed and assembled one for testing last week. Been testing its features for several days now. I’m putting it in a nice IP66-rated enclosure for more tests and development.

This device is meant to be a one-stop-shop for data logging needs with minimal setup and learning time while my SDI-12 USB adapters are meant to add SDI-12 and other sensors to your existing datalogging PC or Raspberry Pi (assuming you’re willing to learn some Linux and the Pi).

For now, here are a couple of photos of just the board:

Board in my hand

Board with a 4-channel hi-res analog input extension board

Here is a list of features:

  • Open-source full-feature Python data logging script
  • SDI-12 translator with 4 SDI-12 sensor connectors (more can be added)
  • 4 analog auto-scaling single-ended analog inputs up to 20uV precision
  • 2 analog auto-scaling differential analog inputs up to 20uV precision (alt. to 4-chn single-end)
  • Socket for optional 2G/3G/4G LTE/ZigBee/Digimesh RF modules
  • Resistance sensing with optional precision 1K or 10K pull-up resistors (0.1% 20ppm/degC)
  • Extension port allows up to 12 more precision analog and resistance sensors
  • Micro-SD card slot (up to 32GB of non-volatile program and data storage)
  • Data file download via FTP (Wi-Fi)
  • Hi-precision temperature-compensated Real Time Clock for time keeping (2-4 ppm)
  • Power cycle SDI-12 sensors to reset problematic SDI-12 sensors
  • Wi-Fi connection to upload data to servers (requires a Wi-Fi hotspot/router)
  • 4 analog single-ended analog inputs up to 5mV precision where less precision is acceptable
  • 4 digital on/off inputs (alt. to 4 5mV analog chn)
  • 4 digital pulse counters for rain gauges, anemometers etc. (alt. to 4 5vV analog chn)
  • Device temperature monitoring
  • Starter MicroSD card with logging script included
  • Xbee sleep to conserve power
  • Interactive sensor and logger setup interface
  • Optional water-proof enclosure
  • Optional wired adapter for more software development

I will add more details later.

SDI-12 USB adapter upgraded

2018-07-02 15.25.47

After some more firmware development and testing, I am happy to announce that the upgraded SDI-12 USB adapter is now available. The above is the first batch of these adapters and one hi-res analog input add-on board.

The upgraded board features the following:

  1. 4 analog inputs. 12-pole terminal block that features 4 analog voltage inputs. These are 10-bit or 5mV resolution inputs without differential reading. There are there to provide basic voltage inputs for projects that don’t require hi-res analog voltage inputs.
  2. Pulse counters. Alternatively, these 4 inputs can be used as pulse counters. Say you have a rain gauge or flow meter that outputs pulses, these pins can count the pulses. You may need additional filtering (one capacitor and two resistors) if the pulses are noisy. Each time you read the pulse counts, you get the counts since you last read and the adapter will start counting from zero again. This way, if you collect data every minute, the counts will be counts/min. Because each data point is accompanied with date-time information, you can always calculate the count rate with your data set.
  3. Extension port. There is now an 8-pin extension port for add-on boards. The first extension board I have designed and tested is a hi-res analog voltage input board. This board features the same four 16-bit auto-scaling inputs and differential inputs as the SDI-12 + Analog USB adapter, with an added benefit of address jumper. You can add as many as 4 such extension boards to the new SDI-12 USB adapter, with each extension board taking a different address. That is up to 16 hi-res voltage inputs.
  4. Serial port. There is also a serial port connector with RX, TX, etc. This port helps you connect the adapter to an arduino or a micropython board that don’t have USB connections but have serial ports.

With the added features, comes added costs of parts, quality checking, and development times. So I am currently offering two-tier pricing:

  1. The board with everything included and tested at $55
  2. The board without the 12-pole terminal block or these pins tested at $45
  3. If you need the serial port, I can solder the header and configure it free of charge.
  4. The hi-res analog add-on board is $35 each. If you want the 4 additional SDI-12 port soldered on it, I can do it for $5 extra.

The full-featured board has the additional terminal block and needs to be tested with all the analog pins to make sure they are all properly connected (reflowed). In the photo, the bread board and 4 blue potentiometer is the test rig I use to test hi-res analog inputs for the SDI-12 + Analog USB adapter. You CAN solder on your own 12-pole terminal block and test the pins yourself too. You will have to do a lot of screw/unscrew of a potentiometer or resistors though. The firmware is the same so once you solder the connector on, you can use the features.

2018-07-02 14.05.33

Although the analog add-on + new adapter costs the same as the original SDI-12 Analog USB adapter, the stack of two boards does add to its height so the board needs more space. My intention is to add flexibility to the adapter so I can later add more features to the ecosystem without scraping the existing devices. I did a custom board for someone that wants magnetometer, accelerometer, and gyroscope with SDI-12 sensors:

2018-06-17 12.15.47.jpg

I glued the sensor board on top of the adapter board and wired them together via the extension header. This way I didn’t have to spend time designing a new board, which will likely cost more time and money. Also, designing a new extension board is easier than designing a whole new adapter with the sensors on it. I can do more custom sensor boards even if there is only market for a few.

The purchase links to the full-featured adapter and the hi-res add-on board have not been established yet. I’ll get them up and running on my blog and at inmojo soon.

New data logging script

multiple sensor and adapter deploymentThe SDI-12 USB adapter data logging script has been updated to version 1.6.0. Here are the new features:

  • New configuration file format. I went from a basic format that is essentially the key strokes of the interactive inputs to a more human-readable, expandable, and future-proof dictionary-type of format.
  • The serial port is remembered in the configuration file by a unique ID instead of device name such as COM3. This makes it impossible to open the wrong serial port. You can also copy your configuration files to a different OS type and expect it to still work.
  • The data file name now includes the configuration file’s name, such as Liudr.cfg_20180709.csv. This makes it possible to distinguish different data that are collected using different configurations.
  • You can specify a different configuration file name in command line instead of the default. This makes it possible to run multiple adapters on the same system, each running its own configuration file. You can also set up different data logging scenarios and store them in different configuration files for easy recall.
  • There are also stability updates that prevent the script from quitting if, say, the sensor fails to return any valid response. Instead, the logger waits until the next data point and try again.
  • The logging script can now use multiple sensing commands per sensor, such as M!, M1!, M2! etc. besides reading multiple sensors. The interactive input has changed a bit to accommodate the upgrade.
  • Now if you run the script for the first time, it takes you to the interactive inputs to collect configuration parameters to save them into a configuration file. Then when you run it the second time, it automatically loads the configuration file. If you wish to make minimal changes, you can just edit the configuration file. If you want to run the interactive inputs again, delete the configuration file and run the script again.

This version is still considered “experimental” while I hunt for potential problems. You can find the script under the download section along with “changes in V1.6.0” document:

https://liudresllc.com/gadget/sdi-12-usb-adapter/

Let me know what you think!

Open FTDI USB-serial UART port by ID

In this post, I will explain how to open serial port to your Arduino or SDI-12 USB adapter by its unique ID so you always open the correct port even when there are multiple such devices on your computer or raspberry pi.

For Arduino and SDI-12 USB adapter users, I have a nice trick to help you manage multiple Arduinos or SDI-12 USB adapters on the same computer or raspberry pi. On raspberry pi, as on a typical linux system, your device shows up as a serial port, such as /dev/ttyUSB0. This serial port designation is usually bound by the order that the device is discovered at boot time, which may not be the same even if you keep your adapter plugged into the same USB port. This means if you have more than one device on your raspberry pi, you may open the wrong port at times, which should be a big issue. To prevent your program from opening the wrong port, you need a unique ID for each device. Luckily FTDI chips already come with unique IDs. We just have to find those IDs and possibly change them into more meaningful things for us to remember. Assume for the moment you are making a data logger for your test fields. There is one field that can be called “NORTH”. The following steps will help you change the ID of the FTDI chip on your device so you can later open its port by that ID, instead of a port name. Here is a list of which devices are using FTDI’s chips that have the reprogrammable ID feature:

  • Liudr SDI-12 USB adapter (all types)
  • Sparkfun Redboard
  • Certain Arduino clone boards
  • Lots of other devices such as GPS etc.

Here is the FT_PROG tool FTDI provides. It’s windows only but I’m sure you can find a windows machine to run it. I’ve not tested it in a virtual machine whose host is linux or macos. I’ll do that when I have more time. If you are unsure whether your device has an FTDI chip, a quick scan using the program will tell you.

http://www.ftdichip.com/Support/Utilities.htm#FT_PROG

First, press the scan icon (magnifying glass). If you have a device with FTDI chip, it will show up. See the screen grab below:

So I have an FT232R chip with a chip serial number “A106DHE5”. I can open port with this serial number but I’d rather change it to “NORTH”. Click on the “SerialNumber” from the left side.

Uncheck the “auto generate serial no” so you can edit the serial number to “NORTH”. You have up to 16 characters to name the adapter. Once done, press flash icon (thunder bolt).

Now that you have programmed your chip, you can read the information back using “scan” again to verify that the ID has changed:

Now that you have this nice ID, let’s open port by this ID.

Here is a small complication. On linux, the chip ID is returned, such as “NORTH”. On window, the port ID is returned, such as “NORTHA“. The addition of the “A” indicates the port “A” on chip “NORTH”. This is because some FTDI chips have two serial ports. The port IDs will be “NORTHA” and “NORTHB“. Even for FTDI chips that have only one port, such as for our case, the “A” is still there. So I recommend comparing chip ID instead of port ID. If you only work on linux/rpi systems, this doesn’t seem to concern you. But if you wish to make your code platform independent, i.e. running on windows without an incident, you will only extract whichever ID you receive with the stored chip ID, up to the length of the stored chip ID. Note: in the platform-independent code, you can’t slice a port’s serial_number with your stored ID because some internal ports don’t have serial numbers thus returns empty that will throw an error when you try to slice an empty array.

The following is a snippet that works ONLY on linux/rpi systems:


[sourcecode language=”python” wraplines=”false” collapse=”false”]
import serial.tools.list_ports # For listing available serial ports
import serial # For serial communication

my_ID=’A817EQLG’
port_device=”
# List ports for user to select
a = serial.tools.list_ports.comports()
print(‘\nDetected the following serial ports:’)
for w in a:
print(‘Port:%s\tID#:=%s’ % (w.device, w.serial_number))
if (w.serial_number==my_ID): # Match ID with the correct port
port_device=w.device # Store the device name to later open port with.
if len(port_device)!=0:
print(‘\r\n%s is the correct port.’ %(port_device))
else:
print("Port with ID: %s is not found!" %(my_ID))
[/sourcecode]

The following is a snippet that works on ALL OS:

[sourcecode language=”python” wraplines=”false” collapse=”false”]
import serial.tools.list_ports # For listing available serial ports
import serial # For serial communication

my_ID=’A817EQLG’
port_device=”
# List ports for user to select
a = serial.tools.list_ports.comports()
print(‘\nDetected the following serial ports:’)
for w in a:
print(‘Port:%s\tID#:=%s’ % (w.device, w.serial_number))
if (w.serial_number.__str__()[:len(my_ID)]==my_ID): # Match ID with the correct port
port_device=w.device # Store the device name to later open port with.
if len(port_device)!=0:
print(‘\r\n%s is the correct port.’ %(port_device))
else:
print("Port with ID: %s is not found!" %(my_ID))
[/sourcecode]

Your choice, simplicity of code or cross-platform compatibility.

Here is the cross-platform code’s result on my windows machine:

Detected the following serial ports:
Port:COM23	ID#:=A817EQLGA
Port:COM3	ID#:=None


COM23 is the correct port.

As you can see, there is an added “A” at the end of the ID reported by windows, which the python code ignored to produce a match.

I’ve also attempted to do this using Processing 3.0. Unfortunately, the Serial.getProperties() function that should return similar information returns blank (possibly not implemented on windows and yet to be tested on linux). If you have tested Processing method with success, please reply below with your results. I’ll add your comment to the post.

Here is the code I used in Processing 3.0:

[sourcecode language=”java” wraplines=”false” collapse=”false”]
import processing.serial.*;
import java.util.Map;

void setup() {
String[] ports = Serial.list();

for (int i=0; i < ports.length; i++) {
Map<String, String> props = Serial.getProperties(ports[i]);
print(ports[i]+": ");
println(props);
}
}
[/sourcecode]

Results:

COM3: {}
COM23: {}
:(

Closing note: even if you work on Windows that assigns unique COM port number to your arduino or adapters, the assignment relies entirely on the currently available port numbers. If you develop your project on one windows PC and deploy on another windows PC, you WILL get different COM port numbers. On a Mac, the ID is embedded on the port name such as /dev/ttl.usbserial-A103RU9T so you are better off. But, will you be willing to shell out the money to get a mac and have it sit somewhere to collect data just because of this feature? If you are a linux wiz, you can bind names with serial numbers using some scripts. That’s beyond the scope of our general discussion, which assumes minimal experience with linux administration.

%d