Open FTDI USB-serial UART port by ID

In this post, I will explain how to open serial port to your Arduino or SDI-12 USB adapter by its unique ID so you always open the correct port even when there are multiple such devices on your computer or raspberry pi.

For Arduino and SDI-12 USB adapter users, I have a nice trick to help you manage multiple Arduinos or SDI-12 USB adapters on the same computer or raspberry pi. On raspberry pi, as on a typical linux system, your device shows up as a serial port, such as /dev/ttyUSB0. This serial port designation is usually bound by the order that the device is discovered at boot time, which may not be the same even if you keep your adapter plugged into the same USB port. This means if you have more than one device on your raspberry pi, you may open the wrong port at times, which should be a big issue. To prevent your program from opening the wrong port, you need a unique ID for each device. Luckily FTDI chips already come with unique IDs. We just have to find those IDs and possibly change them into more meaningful things for us to remember. Assume for the moment you are making a data logger for your test fields. There is one field that can be called “NORTH”. The following steps will help you change the ID of the FTDI chip on your device so you can later open its port by that ID, instead of a port name. Here is a list of which devices are using FTDI’s chips that have the reprogrammable ID feature:

  • Liudr SDI-12 USB adapter (all types)
  • Sparkfun Redboard
  • Certain Arduino clone boards
  • Lots of other devices such as GPS etc.

Here is the FT_PROG tool FTDI provides. It’s windows only but I’m sure you can find a windows machine to run it. I’ve not tested it in a virtual machine whose host is linux or macos. I’ll do that when I have more time. If you are unsure whether your device has an FTDI chip, a quick scan using the program will tell you.

http://www.ftdichip.com/Support/Utilities.htm#FT_PROG

First, press the scan icon (magnifying glass). If you have a device with FTDI chip, it will show up. See the screen grab below:

So I have an FT232R chip with a chip serial number “A106DHE5”. I can open port with this serial number but I’d rather change it to “NORTH”. Click on the “SerialNumber” from the left side.

Uncheck the “auto generate serial no” so you can edit the serial number to “NORTH”. You have up to 16 characters to name the adapter. Once done, press flash icon (thunder bolt).

Now that you have programmed your chip, you can read the information back using “scan” again to verify that the ID has changed:

Now that you have this nice ID, let’s open port by this ID.

Here is a small complication. On linux, the chip ID is returned, such as “NORTH”. On window, the port ID is returned, such as “NORTHA“. The addition of the “A” indicates the port “A” on chip “NORTH”. This is because some FTDI chips have two serial ports. The port IDs will be “NORTHA” and “NORTHB“. Even for FTDI chips that have only one port, such as for our case, the “A” is still there. So I recommend comparing chip ID instead of port ID. If you only work on linux/rpi systems, this doesn’t seem to concern you. But if you wish to make your code platform independent, i.e. running on windows without an incident, you will only extract whichever ID you receive with the stored chip ID, up to the length of the stored chip ID. Note: in the platform-independent code, you can’t slice a port’s serial_number with your stored ID because some internal ports don’t have serial numbers thus returns empty that will throw an error when you try to slice an empty array.

The following is a snippet that works ONLY on linux/rpi systems:


[sourcecode language=”python” wraplines=”false” collapse=”false”]
import serial.tools.list_ports # For listing available serial ports
import serial # For serial communication

my_ID=’A817EQLG’
port_device=”
# List ports for user to select
a = serial.tools.list_ports.comports()
print(‘\nDetected the following serial ports:’)
for w in a:
print(‘Port:%s\tID#:=%s’ % (w.device, w.serial_number))
if (w.serial_number==my_ID): # Match ID with the correct port
port_device=w.device # Store the device name to later open port with.
if len(port_device)!=0:
print(‘\r\n%s is the correct port.’ %(port_device))
else:
print("Port with ID: %s is not found!" %(my_ID))
[/sourcecode]

The following is a snippet that works on ALL OS:

[sourcecode language=”python” wraplines=”false” collapse=”false”]
import serial.tools.list_ports # For listing available serial ports
import serial # For serial communication

my_ID=’A817EQLG’
port_device=”
# List ports for user to select
a = serial.tools.list_ports.comports()
print(‘\nDetected the following serial ports:’)
for w in a:
print(‘Port:%s\tID#:=%s’ % (w.device, w.serial_number))
if (w.serial_number.__str__()[:len(my_ID)]==my_ID): # Match ID with the correct port
port_device=w.device # Store the device name to later open port with.
if len(port_device)!=0:
print(‘\r\n%s is the correct port.’ %(port_device))
else:
print("Port with ID: %s is not found!" %(my_ID))
[/sourcecode]

Your choice, simplicity of code or cross-platform compatibility.

Here is the cross-platform code’s result on my windows machine:

Detected the following serial ports:
Port:COM23	ID#:=A817EQLGA
Port:COM3	ID#:=None


COM23 is the correct port.

As you can see, there is an added “A” at the end of the ID reported by windows, which the python code ignored to produce a match.

I’ve also attempted to do this using Processing 3.0. Unfortunately, the Serial.getProperties() function that should return similar information returns blank (possibly not implemented on windows and yet to be tested on linux). If you have tested Processing method with success, please reply below with your results. I’ll add your comment to the post.

Here is the code I used in Processing 3.0:

[sourcecode language=”java” wraplines=”false” collapse=”false”]
import processing.serial.*;
import java.util.Map;

void setup() {
String[] ports = Serial.list();

for (int i=0; i < ports.length; i++) {
Map<String, String> props = Serial.getProperties(ports[i]);
print(ports[i]+": ");
println(props);
}
}
[/sourcecode]

Results:

COM3: {}
COM23: {}
:(

Closing note: even if you work on Windows that assigns unique COM port number to your arduino or adapters, the assignment relies entirely on the currently available port numbers. If you develop your project on one windows PC and deploy on another windows PC, you WILL get different COM port numbers. On a Mac, the ID is embedded on the port name such as /dev/ttl.usbserial-A103RU9T so you are better off. But, will you be willing to shell out the money to get a mac and have it sit somewhere to collect data just because of this feature? If you are a linux wiz, you can bind names with serial numbers using some scripts. That’s beyond the scope of our general discussion, which assumes minimal experience with linux administration.

Careful with Python indentation

If you are like me, using the default Python integrated development environment, IDLE, you probably feel like me, desiring more features, such as line numbers, horizontal scrolling, code folding, etc. I am aware of a Python module called IDLEX, which has lots of these features. On the other hand, if you are already familiar with text editors such as Notepad ++ (Npp in short), which is excellent, you may want to just switch over to Npp for your Python script editing. You can either use their default color theme or edit it to look like IDLE. There is an essential feature you need to set in order to produce correct indentation though.

Because unlike other major programming languages such as C/C++, Java, etc. Python uses indentation to indicate structure. If you write an if-statement, you do this:

[sourcecode language=”python”]

if (statement==True):
print(‘It is true!’)
else:
print(‘It is not true!’)

[/sourcecode]

I’ve learned a lesson of mixing tab characters and white spaces as indentation. It’s a bad mix. Although I love the tab character more than spaces, I consider spaces better in Python since it is the Python standard. Here is some tab vs. space discussion. They each have advantages. So how do we set up Npp to produce the Python standard 4-spaces whenever we press the TAB key on the keyboard? Here it is:

This “Preferences” dialog is under “Settings” menu. Just make sure you select “Language” on the left and then “Python” on the right (not the middle), deselect “use default value”, and select “Replace by space”. This produces Python 4-space indentations. Once you’re done, you may want to inspect your exist script to see if there are any tabs by selecting option “Show all characters” under “View” menu’s “Show Symbol” option. This way tab characters look like red arrows while white spaces look as red dots.

Now you are using Npp for editing, you can create a “Run” command inside Npp to run your Python code:

Under “Run” menu “Run…” option, you can type up the command python -i “$(FULL_CURRENT_PATH)”

Save a short cut such as Shift + F5. Then you can run the code with the shortcut.

Can’t upgrade pyserial in latest raspbian distribution?

This is just for your information if you are a Raspberry Pi user and playing with Python code from my blog. If you are trying to use the latest distro of raspbian with pyserial for some serial port project, you may have come across this issue that regardless how you upgrade pyserial using pip3, your python3 will always call up the old pyserial 2.6 that came with the distribution. I am a bit disappointed that the foundation has included such an old version of pyserial, couldn’t they just try a pyserial 3.0 instead? My solution was to remove the python3-serial module using apt-get and then install pyserial 3.3 using pip3.

[code language=”bash”]
sudo apt-get remove python3-serial
sudo pip3 install pyserial
[/code]

Hope this helps.

Python code for multiple SDI-12 sensors

As you probably know, the SDI-12 sensor logger code in Python can only log one sensor at a time. It is not a hardware limitation. I wrote the logger code as an example of how to do logging with the SDI-12 adapters and Python. To make sure people don’t have the wrong ideas that you can ONLY get one sensor logged, I have been working on the logger code for the past couple of days and have increased the number of sensors from one to any number you need. The improvement is backward compatible with the configuration file for Raspberry Pi logging, in case you wonder. All that is changed to the user interface is the prompt:

Original prompt:

‘SDI-12 sensor address: (0-9, A-Z, a-z)’

New prompt:

‘Enter all SDI-12 sensor addresses, such as 1234:’

 

So if you have 4 sensors you want to log together, then just enter all their addresses in a string, such as 1234 and hit enter. All sensor inputs will be saved to log file and sent to sparkfun’s data server. The only limitation on the code now is the sparkfun data server stream. The server stream is set up to only take 6 values so the logger code will send the first 6 values from all sensors to the server. If you wish to lift this limitation, you should create your own stream and set up as many values per data point as you need, and modify the logger code (see the magic number 6?).

Below are some sample data logs:

2/3/2017  12:15:25 AM 1 1.11 26 z 5.09419 5.09381 0.24388 5.09419
2/3/2017  12:15:56 AM 1 1.11 26 z 5.09325 5.0925 0.24388 5.09306
2/3/2017  12:16:28 AM 1 1.11 26 z 5.09363 5.094 0.24375 5.09438
2/3/2017  12:17:02 AM 1 1.11 26 z 5.09194 5.09269 0.24375 5.09306

As you can see, the data are separated by sensor address. The address z is the analog-to-digital converter’s address for SDI-12 + Analog adapter. As you can see, my computer outputs 5.09V instead of the nominal 5V on its USB port.

Here is a link to the new logger code. Give it a try and let me know how you like it.

sdi_12_logger_v1_4_1.py

SDI-12 bus scan code

In case you need to diagnose your SDI-12 data logger, I posted SDI-12 bus scan code for the SDI-12 USB adapter (in Python) and for the SDI-12 data logging shield (in C for Arduino). Their links are under Data logger programs or Downloads.

Here is a screen shot of the Python code:

bus-scanner

The Arduino code has a similar interface without the serial port select (you select Arduino serial port in Arduino serial monitor). It’s fun translating C code into Python. C is famous for manipulating strings as arrays of ASCII characters. Doing such in Python seems like a hassle because it has no pointer mechanism, loose types, and aims to handle Unicode so it buries the ASCII characters under layers of things. Anyway, each language has its own strengths and weaknesses.

Notice that the scanner shows a Decagon 5TM sensor at address ‘1’ and also shows the SDU-12 translator at address ‘z’. Make sure that each sensor already has a unique address before scanning the bus. To configure sensors with unique addresses, run the configuration code for either the SDI-12 USB adapter or the SDI-12 data logging shield. Hope this helps. Comments?

Log data with SDI-12 USB adapter

On my last post, I showed a couple of videos of how to connect an SDI-12 sensor to the SDI-12 USB adapter. Here is a video of how to configure an SDI-12 sensor and log data with the adapter:

 

Sorry there was some noise in the background that I couldn’t get rid off with the authoring software. Here is the transcript in case you need it. I will explain how to send data to sparkfun’s data server in my next post.

Let’s plug in the SDI-12 USB adapter. Windows will automatically install its driver and create a COM port, in this case, COM18. If this is your first time installing a COM port on your computer, you will need to be patient.

What we are looking at are two windows. The window on the left is the python shell. It shows you the input and output of python scripts. The window on the right side is the SDI-12 configuration script version 1.1. We run this script to set up the sensor’s SDI-12 address.

On the left window, you can see the messages printed by the configuration script in blue. This script runs on windows, mac, linux, and raspberry pi.

The script has discovered a number of COM ports and listed them all on the left window. The first one, item zero, is our SDI-12 USB adapter, i.e. COM18. Type zero and enter. Ignore the rest. If you are unsure about the COM port number, run this script with the adapter disconnected and then run it again with the adapter connected. This way you can see which port belongs to the adapter.

After a short moment, the script has detected the SDI-12 sensor at address 1. The information printed out indicates sensor address 1, compliant with SDI-12 standard version 1.3, and the manufacturer is Decagon. The sensor is a spectral reflectance sensor and its serial number is also printed out.

What this script does is to detect the one-character SDI-12 sensor address, print out the sensor information, and allows you to change its SDI-12 sensor address. Valid address includes 0-9, A-Z, and a-z.

Let’s set the sensor address to 2. To check that this has taken effect, we run the script again.

Now it has detected the new address. The address is saved on the sensor until it is changed again.

Now let’s look at the data logger script.

What this script does is: it logs data to two places, a file on the computer, and it also sends the same data to a server at sparkfun electronics. Everyone that runs this sample data logging script shares the same storage on the server and can see results from everyone else. You can also create your own storage or stream on sparkfun so you can keep the data to yourself.

Here I have python shell and sdi_12_logger script version 1.1. Let’s run the script with F5. This time there is a longer printout and it does say it runs on Mac OSX.

Next we see a list of serial ports. We select zero for COM18, like before.

Then we are required to enter the total number of data points. Let’s try 5. You can enter any large number of data points and stop the data acquisition anytime with Ctrl-C. Data saved on your computer and sent to the server will not be lost if you stop the script.

Delay between data points is specified in seconds. We’ll use 10 seconds. Then we decide whether to store each data point with local or universal time. Enter 1 to pick local time.

Enter the SDI-12 sensor address, 2.

Here is the first data point. Date and time, then two spectral reflectance values, and then 2.0 for facing upward.

The curl command that appears on the next line will only appear if you have curl installed, which is a tool to send HTTP requests. So besides a local copy of your data, you also have an online copy on sparkfun electronics server.

OK now we finished collecting 5 data points and the python shell prompt has returned.

We will take a look at the data file and the server data next.

Install python on windows PC

Last time I described how to install python on Debian or Raspberry Pi. If you are a beginner and have a windows PC, here is my short tutorial on how to install python AND pyserial on a windows PC. Since I’m interested in Arduino, I will certainly need a serial module in python to talk to Arduino, such as sending Real time (yyyy/mm/dd hh:mm:ss or epoch) to Arduino to set its Real Time Clock or collect data from Arduino.

Python as of the writing has two main versions, Python 2.x and 3.x. The latest 2.x is 2.7.11. Lots of code have been developed on Python 2.x so many people still hang on to this version. The other day I tinkered with a Raspberry Pi LCD hat (add-on) and the Adafruit Python module that drives it can only be installed on Python 2.7.11. The most current version of Python 3 is 3.5.1. We will focus on this version since it has been out for quite a while. Also, Pyserial works flawlessly on Python 3.5.1.

First, download the installer here:

https://www.python.org/downloads/release/python-351/

Go to the bottom of the page and find your PC’s version (32 or 64 bit).

Install Python. Now, you need to find where the installed Python resides so you can include its path in your system’s PATH environment variable.

For me, it is here:

C:\Users\your_username\AppData\Local\Programs\Python\Python35\

Make sure you replace your_username with your actual user name (usually shortened from your full user name in windows 10).

Right click your “This PC” icon on desktop and select “Properties”. Then go to “Advanced system settings” on the left pane.

Once the dialog pops up, select “Environment Variables” on bottom right:

Advanced system settings

Once opened, you see the top half as user variables.

Environment VariablesClick PATH and Edit. Add the following TWO paths:

C:\Users\your_username\AppData\Local\Programs\Python\Python35\

C:\Users\your_username\AppData\Local\Programs\Python\Python35\Scripts\

Save and restart your computer.

Now you should have Python installed and ready to go. Still, you will need to install various modules to expand your Python, or simply to run “that script that wants this and that module”. Lots of modules are available to Python such as pyserial (serial port support) and idlex (more customization to Python’s IDE).

To install pyserial (or any other module), first open a command prompt. Then type in:

“pip install pyserial”

The package will install automatically. You are done!

 

30-day temperature data

Since I was going away for a month, I decided, before I left, to set my SDI-12 data logger to run a 30-day data collection routine, just to see how robust the software is. Here is the result:

30-day temperature

The temperature sensor was in my office so its variation was small. The steep slope towards the end was because I opened the window after I got back and kept the window open for a whole day. You may notice the lack of variation of temperature between 5/10 and 5/13 but that WAS what happened. History data from wunderground.com showed less-than-average variation of outdoor temperature in my area during the same period:

wundergroundSo my conclusion was that my python data logger code running on raspberry pi was robust enough for at least 30 days so was the SDI-12 USB adapter. If this were done outdoors, then the raspberry pi and the SID-12 USB adapter both need protection from the element. Also a solar panel and battery will be needed.

Parts and components:

Raspberry pi 2 B (or 3 B)

SDI-12 USB adapter (Free python data logger code)

Decagon 5TM soil sensor

 

Soil data logger telemetry

I have finally found time to build a simple website for my soil data logger with telemetry. The system works as the following:

  1. The data logger consists of a raspberry pi and my SDI-12 USB adapter with a Decagon 5TM soil sensor
  2. The data logger runs the open-source datalogger code I wrote in Python to first get parameters from the user (COM port, SDI-12 address, delay etc.), and then collect data, save to a local .CSV file, and then send the same data to sparkfun’s phant server.
  3. I constructed a web interface to plot the data using Google Charts and download .CSV version from sparkfun’s phant server.

Here is a screen shot:

soil logger webpage

I’ve uploaded the webpage to a server with a link below. The sensor is apparently NOT buried in soil so I can easily take the setup and set it up in different places to test its stability.

Link to the website: Link

Update SDI-12 USB adapter firmware

I have just released an update to the SDI-12 USB adapter. This update makes the adapter recover from errors in communication so it won’t hang your data logging process in case a sensor is broken while you are logging.

To make the firmware update easy, I wrote a Python script to use avrdude.exe to load the firmware to the adapter. You will run the script just like the data logging script or config script:

  1. Install Python 3.5 with Pyserial 3.0 (instruction in the manual).
  2. Unzip the content of the firmware update package in a folder.
  3. Check the properties of avrdude.exe to make sure that it is not blocked from running.
  4. Run the script SDI_12_firmware_update.py in Python environment IDLE.
  5. Select the adapter’s serial port from a list.
  6. Pick the firmware file, usually SDI_12_translator_v1_x.hex (must be stored in the same folder as the script), then wait for it to complete, which takes less than 30 seconds.

Here is the output from Python:

 

sdi-12 USB adapter firmware update

I had the adapter on COM4.

Once updated, try connect to it using a terminal program and send zI! (zee-EYE!) and you will see the response has version 1.2 in it.

(Updated) This script now works on Windows, Mac OSX, GNU/Linux (64-bit) and Raspberry Pi.

%d