How Arduino bootloader works

atmega328P_bootloader

Everyone that uses Arduino can tell you how easy it is to get projects going once they have an Arduino board in hand. Just load up the blink code and press upload. In a matter of seconds, your Arduino LED is blinking ever so confidently. But do you stop and wonder how Arduino receives the blink code? Enter Arduino bootloader. Every Arduino has a bootloader, a small program that is always stored on the Arduino to update the code in the Arduino. It only runs once per reset. It looks for new code to be loaded to the Arduino before starting the existing code.

The bootloader works like this:

On the PC side, to trigger a reset on the RESET pin, the PC (avrdude.exe or GNU/Linux equivalent) opens the serial port to Arduino when your press upload and the code is ready to upload. This causes the Data Transmit Ready (DTR) line of the USB/TTL chip to go LOW. The Arduino board has a capacitor charging circuit that uses this LOW (charging the capacitor) signal to momentarily pull down the RESET line of the ATMEGA328P chip before returning it to HIGH (capacitor charging completes). So Arduino resets each time its serial port is opened.

Upon reset, Arduino enters the bootloader.

The bootloader looks at the source that caused the reset. There are several sources that can cause a reset. If the reset was caused by the RESET pin, then it waits for one second for the PC to send in commands. When it receives valid commands, it will start accepting new Arduino code in HEX format and erase the existing code to load new one. If it doesn’t receive valid commands, it times out after one second and triggers a Watch Dog Timer (WDT) reset.

Once the bootloader runs again it will look at the source of the reset. Once it determines that it was the WDT reset, it immediately jumps to the first line of the actual code. This way if you power up Arduino, it will be able to immediately run your code, instead of waiting in the bootloader to time out 1 second. It’s pretty smart!

The optiboot bootloader is the most recent bootloader used by Arduino UNO, nano, and other boards based on ATMEGA328P and 1284P. It accepts commands or uploads code at 115,200, unlike the previous bootloader that accepts commands or uploads code at 57,600. A number of people have modified this standard bootloader to fit their own needs, such as slower upload speed for a bare-bone system with no crystals and runs at say 100KHz on internal RC oscillator to preserve battery. Others have made Ethernet upload possible. I’ve personally made a bootloader that will refuse to upload code unless a certain EEPROM byte is set to a certain value. This way if the device is a data logger and to be used by a student, the teacher won’t be worried that the student would get “smart” and erase the code. But when the teacher wants to change the code he/she can enter a password in the existing data logger program and unlock the device for upload again.

If you want to deploy a project, such as an artistic installation or a data logger, you can make a bare-bone system without the whole Arduino UNO board. Then you will need an ATMEGA328P chip with preloaded bootloader like this one, and some extra components.

Anyway, if your code doesn’t need to be changed any further, you can decide to get rid of the bootloader altogether. On an Arduino UNO, this will free up 512 bytes of FLASH and the end user of your device won’t have the danger of erasing the code. To load code without a bootloader, you can use Nick Gammon’s HEX uploader. All you need is another Arduino UNO and an SD card (shield format is preferred) and some jumper wires.

Dropbox Python API

Assume that you are building a data logger and need to send your data from your logger to you, one option that will not cost you money or much programming time is to send your data file to your Dropbox. Dropbox is a cloud storage service. It not only provides you free cloud storage, but also provides application programming interface so you can upload files via programs. This requires some minimal setup. Once set up, you can proceed to add the file upload feature to your code. I am using Python to do the job. It is quick and easy. Plus, you can port your code to any operating system, such as PC running windows, GNU/Linux, Mac OSX, or Raspberry Pi. I am assuming that you have Python 3.X.

There are two versions of Dropbox Python API, V1 and V2. V2 came out around the end of 2015 and only has minimal tutorial on Dropbox.com. Nevertheless, I will use V2. Installing the API is a snap:

On Windows:

Start a command prompt and enter:

pip install dropbox

On Raspberry Pi (Raspbian Jessie) or Debian PC

sudo pip3 install dropbox

If you still have Raspbian Wheezy, pip3 may tell you that there is a newer version of pip. Don’t attempt to upgrade your pip or pip3. It will break.

Here is a short Python script to upload and download files:

[code language=”python”]
import dropbox
file_name=’test_image.jpg’
dropbox_path=’/’
dbx=dropbox.Dropbox(‘Your access token’)
with open(file_name, ‘rb’) as f:
    dbx.files_upload(f.read(),dropbox_path+file_name,mute=True) # The change from f to f.read() was made to comply with the late-2016 change in the API.

dbx.files_download_to_file(‘Copy of ‘+file_name,dropbox_path+file_name)
[/code]

This was easy. There are only three functions that I used, first an authentication, then upload, followed by download. Your access token is generated by Dropbox. This post has the details.
The function dropbox.Dropbox() returns an object. You can use this object to upload or download files and more.
The files_upload() function does the upload. You need to first open the file on your local computer with open(). I am assuming the file is stored in the same folder as your Python script. This returns a file handler f. Then pass f to the files_upload(). This is the first argument. The next argument is the path you want the file to be uploaded to dropbox. It has to start with ‘/’, then the file name (including additional path). You don’t have to preserve file name. The third argument is useful. When you sent mute to True, you won’t get notification for the file upload. If you are making a camera trap, you don’t want your PC flooded with Dropbox notifications just because a bunny decides to visit your backyard.
The files_download_to_file requires first a local computer name, then the Dropbox file name. Again you don’t have to preserve file name. It’s useful to first check if the file exists on your local computer and decide what to do (overwrite, or add prefix/postfix to new file).
This is it! How you use it is up to you.

If you are interested, you can explore the rest of the API, such as creating folders, moving files, listing folders, etc. by reading the documentation (alert: document is very dry)

https://www.dropbox.com/developers/documentation/python

In my next post, I will make a simple Raspberry Pi camera logger to activate only during several intervals of time of the day.

Connect to Dropbox

If you are making a data logger, or camera trap, this is definitely good news. With the right approach, you can sit in your home and data/pictures just flow to your desktop. All you need is internet connection and some Dropbox programming. With a raspberry pi or a pc, you can have your data logger or camera logger automatically upload data to your Dropbox. On a windows PC, this is nothing more than saving your data file to a folder inside your Dropbox folder. Still, constantly writing to a Drpobox file is not the best way to make use of it. You won’t be able to have a meaningful history of the file since the your data logger updates the file too many times. Also, raspberry pi doesn’t have dropbox client. So it makes sense to programmatically upload your file to Dropbox using its API or application programming interface.

I am starting a series or posts to detail how to upload files to Dropbox from within your code. This post only explains how to set things up.

Using Python, it is extremely easy to upload files to dropbox. There is only one catch, you have to set up as an app developer. Here is their logic thus how to get set up:

You have some good ideas for an app. You want to store your app’s data files on dropbox so your user can access them on any device they are using. Good idea, but how to set it up?

First you need a Dropbox account yourself. Once you’re set, log on with your web browser. You’ll see three circles on the bottom left of your window:

dropbox dev-1

Clicking the dots brings up a menu. Select Developers:

dropbox dev-2

 

Click “Create app” to create a new app. I’ve already created a couple of similar apps. It looks like I’ve been trying this camera logger idea for one too many times 😀

dropbox dev-3

Make sure that you select the right options. Select “App folder” access so all your actions are contained in a folder inside the App folder. It used to be called sandbox access. You may not want this app to access your entire Dropbox folder and accidentally delete files it is not intended to touch.

dropbox dev-4

Once done, you will be able to edit your project. Scroll down to find “Generated access token”. This will give you a very long alpha-numeric sequence that grants access to your Dropbox app folder, or the entire Dropbox if you chose “Full Dropbox”. Safeguard this string. If you need to ask questions about your code, remove/black out this string. Anyone with this string can do anything they want with your Dropbox app folder, no password needed. alternatively, you may try the App key and App secret path. That is a longer path than we need to take, just to show how things work.

dropbox dev-5

Now you are all set to develop this app, which is probably a program that uploads data/pictures to your dropbox. Read my next post about how to access Dropbox using Python. Python is one of the least preferred language I use but there are benefits of using it. It’s quick to see results. I am a big fan of Arduino. Unfortunately, it is almost impossible to upload to Dropbox using just an Arduino (except Arduino Yun).

SDI-12 USB adapter on Raspberry pi

[UPDATE] Please follow this link to the SDI-12 USB adapter page for up-to-date information:

https://liudresllc.com/gadget/sdi-12-usb-adapter/

I have successfully developed Python code to run the SDI-12 USB adapter data logger on Raspberry PI. It was quite a learning experience for me but I was able to boil down the steps into a tutorial on how to install the latest Python on Debian/Raspbian here.

Here are some screen shots of the data logger:

2016-02-03-171037_1680x1050_scrot

2016-02-05-100346_1680x1050_scrotI was using a Decagon 5TM soil temperature and humidity sensor (hanging free in air and sometimes touching my table).

I have translated the Tera Term scripts into Python so it would run on all operating systems, including Win 10, Linux (Debian), and Raspberry PI (Raspbian). I have not got my Mac fixed but don’t expect any issues.

Now you have two Python scripts:

The configuration script detects the SDI-12 sensor’s address, prints the name of the sensor, and you can change it to a different address.

The data logger script lists all available serial ports to let you choose the correct one, then asks for sensor address, total data points, delays between data points, and also what time (GMT or local) to use. You can expand the script to take data from multiple probes and upload data to a server. I’ll add more info on the server upload to sparkfun’s phant data server.

While developing the code, I also discovered that the serial port console was very useful. You can set up auto login and run the data logger script upon auto login. This way the data logger starts logging data once the RPI boots up and auto logs on. No need to log in or use VNC (not good if your logger is using 4G hotspot for internet). Just a command prompt is all that is needed. Next step is to add a config file so the data logger will no longer ask for user inputs when this file exists and takes parameters from the file. This config file will reside in the FAT partition so that the user can easily update the parameters without having to boot into raspberry pi. Just remove the sd card and change parameters on a PC.

Install python on raspberry pi or debian

I have just started learning the python programming language. It is very different than C/C++ that I’m used to. So to Arduino and Java folks it will take some getting used to. But the reason that I am learning is the promise of cross-platform portability and powerful libraries/modules. If you could send an HTTP request in a few lines, or imagine that you write and test a program on PC and run it on a raspberry pi with no/minimal modification, would you stop complaining about weird syntax or loose data types and just learn the darn language? Well, I did.

So in order to use Python, you need to install it. Since I am interested in using raspberry pi with an arduino, I need the best serial port support that I can find. My conclusion is to use Python 3.5.1 and pyserial 3.0.1 (the latest of both as of the blog post).

On a windows PC, this would be the easiest. Download the installers and install the program.

https://www.python.org/downloads/release/python-351/

https://pypi.python.org/pypi/pyserial

On a linux PC or raspberry pi, python is included but is a lower version. The tutorial helps you install python on linux machines and raspberry pi’s.

This tutorial is based on this tutorial and pieces of other tutorials and information online:

http://www.extellisys.com/articles/python-on-debian-wheezy

To install the latest version of python, you need to build it from source code.

  1. Download the source code and unzip it in a folder.

https://www.python.org/downloads/source/

  1. Update apt-get first

$ sudo apt-get update

  1. Install required tools to build Python from source. Debian/Raspbian Jessie will install libdb5.3-dev while Wheezy will install libdb5.1-dev. You need to install tk-dev to make Python’s IDE IDLE3.5 work

$ sudo apt-get install build-essential

$ sudo apt-get install tk-dev

$ sudo apt-get install libncurses5-dev libncursesw5-dev libreadline6-dev

$ sudo apt-get install libdb5.1-dev libgdbm-dev libsqlite3-dev libssl-dev

$ sudo apt-get install libbz2-dev libexpat1-dev liblzma-dev zlib1g-dev

  1. Prepare for Pip (I didn’t do it but the tutorial I followed mentioned this, which seems a bit old)

$ mkdir -p ~/.pip/cache

$ echo ‘[global]’ > ~/.pip/pip.conf

$ echo ‘download_cache = ~/.pip/cache’ >> ~/.pip/pip.conf

  1. Make python in the unzipped source code folder (Python-3.5.1), install it in a folder that will not overwrite the current python version, such as in /usr/local/opt, delete upzipped source folder after done

$ cd Python-3.5.1

$ ./configure –prefix=/usr/local/opt/python-3.5.1

$ make

$ sudo make install

$ sudo mkdir /usr/local/opt/python-3.5.1

  1. Download pyserial source code and unzip in a folder.
  2. In the unzipped folder, run python 3.5 to install the module. If you used an alias to call python, then the install is in the default python version (3.4 on RPI) so don’t use an alias.

$ sudo /usr/local/opt/python-3.5.1/bin./python3.5 setup.py install

  1. Make aliases to run python 3.5 and idle more easily. You have to log out and back in after making them.

$ echo ‘alias python35=”/usr/local/opt/python3.5.1/bin/python3.5″‘ >> .bashrc

$ echo ‘alias idle35=”/usr/local/opt/python3.5.1/bin/idle3.5″‘ >> .bashrc

Now you are done! I forgot to mention that if you are installing Python on raspberry pi 1 or zero, the build process will take quite some time to complete. Log out and back in. In a terminal, type idle35 and you will be able to run your new Python.

%d