The current states and brief history the SDI-12 USB adapters

Despite the ongoing chip shortage, I’ve secured a batch of processors for more adapters. At the moment I’ve also built enough stock to last for a little while.

So currently I have about 50 qty adapters including a few with analog inputs. I have more boards and ICs but want to conserve my parts. From what I saw, there won’t be a relief of chip shortage until a year from now, i.e. May 2023. I’ve maintained the price of $49 (sometimes $45) for the last decade by absorbing the cost of parts etc. I might be able to continue to do so if I get some help with assembly, such as getting a pick-and-place machine to place surface parts for me. But it’s an expensive machine.

Here is a brief history of my adapters:

In the summer of 2015, I came up with the idea to make a USB adapter that can make reading SDI-12 sensors easier, after having to deal with this protocol in a number of projects. This purple board was the first batch made at oshpark.com. It is a small PCB fab business (they only collect orders and send to actual fab houses) on the west coast. They used to be called dorkbot PDX, a small group of electronics hobbists on the west coast trying to make affordable boards by banding together in their group orders.

First SDI-12 USB adapter prototype in November 2015

This board has the same general layout as the current version, with the miniUSB (maybe my obsession or did I just purchase too many of these connectors?!), USB-UART chip from FTDI, an atmega328p-au, and a programming header. Back then I only had a single terminal for an SDI-12 sensor and had no option to supply external power, well, not easily:

Connecting to external power 2016

Back in the days, I was using a terminal program to talk to the adapter and wrote a macro based of the terminal program to automate data collection. In 2016, I started learning Python more seriously, maybe it’s from raspberry pi, or maybe it was trying to teach physics with some computing elements, or just having to learn it to do projects. So here comes the logging script, in its primitive form in early 2016:

Python script running on Raspberry pi in Feb. 2016

I also started marketing the adapter as compatible with WIN/NIX/MACOS/RPI because back then Python 3.x was able to bring a more consistent programming experience across these systems that Python 2.x was never able to.

Back then, IoT (Internet of Things) was in its very primitive stage and not many places allowed data to be uploaded and presented online. I started with Sparkfun’s Phant server and thingspeak.com, not sure which one first.

Logging data to the internet in April 2016

Later in 2016, I realized there were some need for high precision analog inputs and added the SDI-12 USB + Analog adapter to the lineup. This is the first prototype, in purple as oshpark.com purple. Placing the tiny analog-to-digital adapter used to be nerve wracking. I have since mastered it.

SDI-12 + Analog USB adapter in November 2016

I had a number of interesting projects in 2016-17 and didn’t do a lot on the SDI-12 USB adapters until later in 2017, when I had a project that required collecting GPS data with the sensor so I designed this for the project and general use.

SDI-12 + GPS USB adapter in July 2017

Back then, the basic adapter was still a small half-sized green board with a single SDI-12 terminal. That was able to change:

Full-sized (since then) SDI-12 USB adapter in March 2018

I didn’t market this as a separate adapter because I had high hope that this will replace the more expensive GPS adapter but the hardware didn’t support GPS. So, I started marketing this as a full-sized adapter at the same price point, $49, and gradually retired the half-sized green boards with single terminals. Starting with this version, I added external power connection and a jumper to select between 5V from USB, where most sensors work fine, and the external power terminal, where you can supply your own DC voltage such as 12V. This feature came from the Analog and GPS adapters.

Later in 2018, I redesigned the basic adapter to mostly today’s look and feel. It now has the 4 terminals, power input and selector on the left and right sides, optional basic analog and digital inputs on top, and an extension port in the middle, with an option of UART instead of USB as well.

Very recognizable look of SDI-12 USB adapter in May 2018

Here is a look at two high resolution analog extension boards on top of the basic adapter. I completed updating the code to take inputs and auto scale the inputs, for up to 4 such extension boards:

High resolution analog input extension boards May 2018

I also added extra SDI-12 terminals extension board to help manage the many wires you may have to manage if you want to connect more sensors.

I have also started customizing my adapters to connect to other sensors, such as the following with accelerometers, from special requests.

Customized adapter with accelerometers June 2018

Later that summer, I designed my own data logger! This was based on the idea of having an embedded SDI-12 processing unit, an analog input, and then an ESP32 processor running a new thing called microPython, new and getting traction back then, both the processor and software, now they’ve become major hardware and software players, if not dominating the IoT. This logger is very promising, with WiFi, Xbee, possibility of 4G LTE-M, sd card, real-time clock, and lots of expension!

My own complete WiFi Xbee GSM etc. data logger July 2018

Lots of things have been happening back then and I did a few rounds of firmware development but didn’t end up releasing this logger! Some of you may know that the spirit of this logger lives on and is thriving in a different sector of the industry! The following look may be familiar?

My logger in an SK-16 enclosure August 2018

In early 2019, I added the UART interface adapter to the lineup, based off the same board as USB interfaced adapter, due to an increase of demand to interface with Arduino, ESP32 etc.

UART adapter March 2019

In June, I contemplated using larger terminals based on user feedback. I decided, in order to maintain the same size and look, I can use 0.175″ pitch terminals instead of 0.1″. This makes the terminals accept thicker wires and separate the wires further for installation and prevent short circuits.

Comparison of 0.1″ and 0.175″ pitch terminals July 2019

Now this looks really really like what I’m selling today:

SDI-12 USB Adapter July 2019

I have also been logging data from my own back yard to demonstrate and test my logging scripts:

My backyard sensor run in the summer of 2019

I was even able to determine that after heavy rainfalls I always had power outage on my logger (see long flat blue lines after spikes). It turned out to be a faulty outdoor outlet in my house tripping up the circuit breaker after heavy rain water seeped into it causing short circuit. I later fixed it:

Bad outdoor outlet I replaced in 2019
Good one I installed in 2019

Lots of things were happening later in 2019, besides what you all know, so I didn’t touch my designs until MUCH later, in early 2021. I added a protection diode to fight 12V accidentally damaging the adapter by operators’ errors.

Adapter with protection diode February 2021

As many of use know, IC shortage soon hit the main street manufacturing. Here was a screen grab of the long way, from 2021:

Chip shortage starting showing long lead times, over 1 year, June 2021

Luckily I was well stocked at the year end of 2020! Now it has become a constant thing in the back of my mind, find parts and stock up!

Showing my stock of parts in 2021 to weather the “temporary” shortage in 2021

Over the years, I got a lot of good suggestions from users. Here is one suggestion to try and read data from a phone. This only works on an android and requires a wire between the adapter and the phone, but it’s a good start.

I started experimenting with android phone and my adapter July 2021

Here is another request from users to handle more sensors with easier terminals and combat the environments. I designed this to fit inside a specific enclosure, the same one I used for my own logger, which was now 3-4 yr old and didn’t get an update (I did update it but didn’t build a prototype). I started selling this to a research group for their projects but held back because I didn’t have many chips left (and it may be a lookalike to something else too)!

SDI-12 USB SK-16 adapter February 2022

I was literally counting my chips at this point! Fortunately I found a small batch of the processor in a smaller form factor and decided to take a risk with the supplier. So now we have a new revision. The protection diode has been replaced by a surface mount version but I’ve kept the thruhole part footprint because I bought a batch of the thruhole diodes, just in case. So if you see your board has an orange diode, or no such diode but a small black box on the top right, both are fine!

New design with smaller processor April 2022

The smaller processor takes quite a bit more time to place and I always place it before all other parts so I can turn the board around to see if I placed it perfectly or not. There aren’t any pins on the chip so it’s harder to check. Hopefully when I get a pick and place machine, I don’t have to worry about it.

Size comparison between the two processors April 2022

So there you have it! I forgot I was planning for a brief history. As a closing remark, I’ve seen the gradual shift of users of this adapter from almost entirely academic research to an increasing percentage of applications to more individual users. What’s not changed is that people love simple and inexpensive solutions to their data logging needs and I’m proud that my adapters have met their needs so far! Thanks for your time!

SDI-12 USB adapter family portrait May 2022

Sentek SDI-12 soil probe troubleshooting

A number of years ago, I had an opportunity to use a Sentek SDI-12 soil probe with 16 sensor nodes in the probe. It was quite something! Lots of data to extract and lots of measurement and data commands to issue. I think that my experience with the probe helped me better understand SDI-12 protocol and ultimately helped me develop and test my SDI-12 USB adapter’s firmware. Recently one of my customers reached out to me regarding troubleshooting tips with this sensor. I felt quite interested to help and refresh my memory on how to properly use the sensor. Here are the troubleshooting tips in case you need them.

First of all, the sensor probe has multiple sensor nodes, making it require longer delays before data can be extracted from it. According to the manual, if you have 16 sensor nodes, the complete moisture measurement M! could take up to 13 seconds (first 9 sensors in the probe), then M1! could take up to 11 seconds (next 7 sensors in the probe). Salinity takes up to 23 seconds with M2! (first 9 sensors in the probe), then M3! could take up to 18 seconds (next 7 sensors in the probe). Temperature and humidity measurements are faster so they only take up to 5 seconds for all 16 sensors.

So if you have more than 9 sensor nodes, you need all measurement commands M! (AKA M0!) thru M7!. If you have 9 sensor nodes or less, you only need M!, M2!, M4!, and M6!.

What this means to you is, you must use my logging script version 1.6.x, which allows you to enter multiple measurement commands per sensor address. For instance, you have 16 sensor nodes, you want all measurements, you can enter 01234567 when asked what measurement commands to use. The ‘0’ means the M! command, which is also known as M0!. Then ‘1’ means M1! command. If you only have 9 or less sensors and you want all data, you need to enter 0246 for all measurements. But if you only want moisture and temperature nothing else, you would use 04.

Next, you want your delay between data points to be longer than these values. If you wish to save ALL data every minute, and you have 16 sensor nodes, this may be too little delay. You can try out delay between data yourself. If you see -999.999, then you need to increase your delay.

Finally, because more sensor nodes require more time before data become available, you may have to increase the serial port timeout value from 10 to a larger value, if you’re not able to obtain data from your sensor.

ser.append(serial.Serial(port=port_device, baudrate=9600, timeout=10))

This is the line you need to change the timeout. The script only waits this long after issuing a measurement command before it times out. Increasing this value will NOT slow down your data collection. The time your sensor requires to get data determines how much time is need to get your data.

Upload data to ThingSpeak

I have been using sparkfun’s Phant server for data upload and retrieval for a few years since they started the service. The service was easy to use and was free. Several months back they discontinued the service, unfortunately. I started looking for suitable alternatives.

Sparkfun recommended three, ThingSpeak, Cayenne, and Blynk. I went ahead and did some research on these services. My goal is to be able to log data online and later retrieve them and possibly visualize them using google charts, like before. I am not at the moment interested in automating my home with actuators or smart phone apps to turn on and off my hall lights. Here are my findings and why I decided that ThingSpeak was the best fit. If you are logging data for later processing or visualization, read on.

ThingSpeak.com

This service is provided by the company behind Matlab. I am not a fan of expensive commercial data manipulation tools such as Matlab but they do have a fair amount of business between universities and industry so their service might be a safer way to go against sudden discontinuation of service such as sparkfun. Basically you create a data stream and send data to the stream. It’s very similar to sparkfun. You can also retrieve your data, possibly good for running your data through google chart. They also provide some basic graphs and matlab analysis tools that I have yet tried.

There are two types of application programming interfaces (APIs) you can use: a REST API, and an MQTT API.

The REST API is based on HTTP so it’s very similar to existing services elsewhere. You use HTTP GET or POST command to send data using an API key, like a private key with sparkfun. Your data are limited to up eight values per data point. If you need more, then you need to create more streams. They also have a bulk update feature that you can use to send multiple sets of data instead of one set of data. This method allows a device to collect data and sleep in between data points. Then when it collects a fair amount of data, it connects to the Internet and sends all data in one shot. It saves power and network bandwidth. You can also create and modify the properties of streams with this API.

With the MQTT API, the underlying protocol is TCP/IP. There is no acknowledgement of data received and it is intended for low-powered devices to just wake up, take data, send it out, and go back to sleep. From my tests, data sent via this method were lost over 50% of the time. Unless future holds differently, I am not recommending this API.

Getting start is easy with ThingSpeak. Just set up an account and follow their tutorial to create a new data stream. Then the following bash code should get you started posting code:

[code language=”bash”]

curl "https://api.thingspeak.com/update/?api_key=your_appkey&field1=1.23&field2=2.34"

[/code]

You can post any number of field values between 1 and 8. You will receive a zero as a positive response. Then you will see your results like the plot on the top of this page.

I have updated my Python data logging code to use ThingSpeak.com instead of the now discontinued data.sparkfun.com. The plot in this post is from Thingspeak.com. Here is a link to the data stream:

https://thingspeak.com/channels/359964

SDI-12 + GPS USB adapter

After a final revision, I am happy to release the SDI-12 GPS USB adapter! This adapter is the latest one to add to the line of SDI-12 USB adapters. In August 2015, I released my first SDI-12 USB adapter with this post. It was an idea that I thought about while traveling. I was working on data logger designs that use SDI-12 sensors and felt that interacting with SDI-12 sensors is not easy for agricultural or water resource researchers. Having an adapter that connects a computer to an SDI-12 sensor and reads measurements directly from the sensor would be very useful. So I made the adapter to simplify lab tests and data logger deployments. Since then, I’ve written free Python scripts for basic data logging (read the SDI-12 USB adapter main page). The demand for the adapter since then has been high enough to support my continued update on the data logging script, expanding from PC/Mac/Linux to single-board computers such as Raspberry Pi and Beagle Bone Bone. I have also expanded the adapter with an SDI-12 + Analog USB adapter that includes four high-precision analog inputs.

Later I found some need to add GPS modules to the existing SDI-12 USB adapter so that mobile data loggers such as those mounted on tractors will be able to produce with Geo-tagged data that can be made into maps. After some initial struggle using the new ATMEGA328PB processor that sports two hardware serial ports (one to talk to PC and the other with GPS), I realized that the GPS module actually interfered with the processor and caused program freeze-up. Then I made some hardware revisions and was able to prevent interference. It turned out that the new ATMEGA328PB processor that I used in my initial prototype was especially susceptible to interference when I used its second hardware serial port that have the same pins as the SPI pins that program the processor. So I switched to the ATMEGA1284P processor that I have been using on my open source physics laboratory design.

After extensive tests, I am happy to add this adapter to the product line. You can purchase (small quantity at the moment) at inmojo.com or on my blog (in the middle of the page). The adapter requires a separate purchase of the GPS module that Adafruit makes and sells, the Ultimate GPS module part number 746. You only need to solder four pins on the GPS module, the TX, RX, GND, and VIN, and the same pins on the adapter. Since the GPS module is relatively expensive, I can’t stock them up. But if you really need it assembled, you may have a GPS unit sent to me and a few extra dollars for assembly and testing. Just contact me once you make a purchase if you want assembly.

Open source data logger

I have been designing data logger for a number of years. This is my answer to lots of data logging needs. An Arduino Nano-based open source data logger:

ospl-th-on

The logger provides the following features (in green) including features of Arduino Nano (in black):

Microcontroller Atmel ATMEGA328P
Power 5 V via USB or 2X AA battery (internally)
Digital I/O 10 (4 PWM output, other Arduino pins used internally)
Analog Input 4 10-bit ADC (8 on ATMEGA328P, only 4 brought out)
DC Current per I/O Pin 40 mA max
Flash Memory 32 KB of which 2 KB used by bootloader
SRAM 2 KB
EEPROM 1 KB on ATMEGA328P, 32 KB on real-time clock breakout board
Clock Speed 16 MHz
MicroSD card 32 GB maximum
Real-time clock Temperature compensated (DS3231)
ADS1115 4-chn 16-bit differential ADC with up to 16X programmable gain
LCD 16 column by 2 row character LCD with back light on/off control
Input Rotary encoder with switch (when shaft is pressed)

Table. Specification of Arduino Nano and the rest of the modules.

Another photo:

red-version-assembled-lcd-removed

As you can see, the logger incorporates a number of breakout boards instead of including these ICs on a single circuit board. More to come…

Phi-shield revised and released

phi-3-shield-on-in-hand

It has been a while since I gave the phi-shield a major revision. I’ve been working on this for a while and now I am releasing the Phi-3 shield. This shield continues to support user interaction with LCDs and buttons. Here is a list of the features:

The following hardware are provided by the shield:

  • 20X4 LCD with back light on/off control
  • Six buttons (up/down/left/right/B/A)
  • Two LED indicators
  • Speaker
  • MicroSD card slot
  • Real-time clock (DS3231)
  • EEPROM (32KB 24LC256)
  • Connector for Adafruit Ultimate GPS module or Bluetooth module
  • Stacking headers for easy access to all pins.
  • Recessed board right edge for easy access to MEGA’s 18X2 pin headers on the right side.
  • Reset button

phi-3-shield-lcd-side-by-side

The following software functions are provided by various supporting libraries:

  • User-selectable menu (LCD + buttons)
  • Number and text entry (LCD + buttons)
  • Scrollable long text (LCD + buttons)
  • Date and time (DS3231 or GPS)
  • Location (GPS)
  • Data and configuration storage (MicroSD card and EEPROM)
  • Playing simple tones (speaker)
  • Indicators (LEDs)
  • Wireless connection (Bluetooth module)

phi-3-shield-lcd-removed-annotated

There are three tiers of Phi-3 shield kits: kit0, kit1, and kit2, none of which includes a GPS module. The kits are immediately available. Buttons with color caps as pictured will be included while supplies last.

Here is the Phi-3 shield’s own page. There are links on the page to make purchases. Or you can visit the BUY page to see what stores carry this shield.

Phi-3 shield

Video demonstrations will be available next week. Meanwhile, the support of Phi-2 shield will remain. If you need Phi-2 shields, I have them available.

phi-3-shield-bottom-rtc-lcd-wire-removed

Python code for multiple SDI-12 sensors

As you probably know, the SDI-12 sensor logger code in Python can only log one sensor at a time. It is not a hardware limitation. I wrote the logger code as an example of how to do logging with the SDI-12 adapters and Python. To make sure people don’t have the wrong ideas that you can ONLY get one sensor logged, I have been working on the logger code for the past couple of days and have increased the number of sensors from one to any number you need. The improvement is backward compatible with the configuration file for Raspberry Pi logging, in case you wonder. All that is changed to the user interface is the prompt:

Original prompt:

‘SDI-12 sensor address: (0-9, A-Z, a-z)’

New prompt:

‘Enter all SDI-12 sensor addresses, such as 1234:’

 

So if you have 4 sensors you want to log together, then just enter all their addresses in a string, such as 1234 and hit enter. All sensor inputs will be saved to log file and sent to sparkfun’s data server. The only limitation on the code now is the sparkfun data server stream. The server stream is set up to only take 6 values so the logger code will send the first 6 values from all sensors to the server. If you wish to lift this limitation, you should create your own stream and set up as many values per data point as you need, and modify the logger code (see the magic number 6?).

Below are some sample data logs:

2/3/2017  12:15:25 AM 1 1.11 26 z 5.09419 5.09381 0.24388 5.09419
2/3/2017  12:15:56 AM 1 1.11 26 z 5.09325 5.0925 0.24388 5.09306
2/3/2017  12:16:28 AM 1 1.11 26 z 5.09363 5.094 0.24375 5.09438
2/3/2017  12:17:02 AM 1 1.11 26 z 5.09194 5.09269 0.24375 5.09306

As you can see, the data are separated by sensor address. The address z is the analog-to-digital converter’s address for SDI-12 + Analog adapter. As you can see, my computer outputs 5.09V instead of the nominal 5V on its USB port.

Here is a link to the new logger code. Give it a try and let me know how you like it.

sdi_12_logger_v1_4_1.py

SDI-12 USB adapter

After some delay, the SDI-12 USB adapter is finally here:

2015-10-03 16.16.57

This adapter is extremely easy to use. Just connect it to your PC and SDI-12 sensor. Then you can use any serial monitor or terminal emulator program to talk with your sensor. Just open the serial port at 9600 Baud rate. You can start by sending device identification command such as ?!. You will see a response from your sensor, which is the one-character address of your sensor. If you have not set its address, it is most likely to be zero (0). Then you can use 0I! to find out the manufacture and model of your sensor, before getting measurements from it. Getting measurement is easy as pie. First send 0M!, then wait for response. Then send 0D0! to fetch the measurements.

For PC users, I even wrote a data logger script that can automatically log data using the popular Tera Term program. You can choose sensor address, total number of data points, delay between points, and time zone when logging, then the program will keep logging data. The following is a screen grab of Tera Term. The sensor is a Decagon 5TM soil temperature and moisture sensor. The address is one (1) and the returned values are relative dielectric permittivity and then temperature in Celsius.

Data logger

Once you get data logging going, you can import the .CSV file into your Excel and plot it. You can choose a proper refresh rate so your data and plot are up to date when you look at them.

Plot

I’m still ordering more circuit boards but should be able to sell these on my inmojo.com store starting now. There is even a quantity discount if you need 10 or more.

Inmojo store sales page

%d