How Arduino bootloader works

February 26, 2016 3 Comments

atmega328P_bootloader

Everyone that uses Arduino can tell you how easy it is to get projects going once they have an Arduino board in hand. Just load up the blink code and press upload. In a matter of seconds, your Arduino LED is blinking ever so confidently. But do you stop and wonder how Arduino receives the blink code? Enter Arduino bootloader. Every Arduino has a bootloader, a small program that is always stored on the Arduino to update the code in the Arduino. It only runs once per reset. It looks for new code to be loaded to the Arduino before starting the existing code.

The bootloader works like this:

On the PC side, to trigger a reset on the RESET pin, the PC (avrdude.exe or GNU/Linux equivalent) opens the serial port to Arduino when your press upload and the code is ready to upload. This causes the Data Transmit Ready (DTR) line of the USB/TTL chip to go LOW. The Arduino board has a capacitor charging circuit that uses this LOW (charging the capacitor) signal to momentarily pull down the RESET line of the ATMEGA328P chip before returning it to HIGH (capacitor charging completes). So Arduino resets each time its serial port is opened.

Upon reset, Arduino enters the bootloader.

The bootloader looks at the source that caused the reset. There are several sources that can cause a reset. If the reset was caused by the RESET pin, then it waits for one second for the PC to send in commands. When it receives valid commands, it will start accepting new Arduino code in HEX format and erase the existing code to load new one. If it doesn’t receive valid commands, it times out after one second and triggers a Watch Dog Timer (WDT) reset.

Once the bootloader runs again it will look at the source of the reset. Once it determines that it was the WDT reset, it immediately jumps to the first line of the actual code. This way if you power up Arduino, it will be able to immediately run your code, instead of waiting in the bootloader to time out 1 second. It’s pretty smart!

The optiboot bootloader is the most recent bootloader used by Arduino UNO, nano, and other boards based on ATMEGA328P and 1284P. It accepts commands or uploads code at 115,200, unlike the previous bootloader that accepts commands or uploads code at 57,600. A number of people have modified this standard bootloader to fit their own needs, such as slower upload speed for a bare-bone system with no crystals and runs at say 100KHz on internal RC oscillator to preserve battery. Others have made Ethernet upload possible. I’ve personally made a bootloader that will refuse to upload code unless a certain EEPROM byte is set to a certain value. This way if the device is a data logger and to be used by a student, the teacher won’t be worried that the student would get “smart” and erase the code. But when the teacher wants to change the code he/she can enter a password in the existing data logger program and unlock the device for upload again.

If you want to deploy a project, such as an artistic installation or a data logger, you can make a bare-bone system without the whole Arduino UNO board. Then you will need an ATMEGA328P chip with preloaded bootloader like this one, and some extra components.

Anyway, if your code doesn’t need to be changed any further, you can decide to get rid of the bootloader altogether. On an Arduino UNO, this will free up 512 bytes of FLASH and the end user of your device won’t have the danger of erasing the code. To load code without a bootloader, you can use Nick Gammon’s HEX uploader. All you need is another Arduino UNO and an SD card (shield format is preferred) and some jumper wires.

Filed under Arduino, Coding

Dropbox Python API

February 20, 2016 6 Comments

Assume that you are building a data logger and need to send your data from your logger to you, one option that will not cost you money or much programming time is to send your data file to your Dropbox. Dropbox is a cloud storage service. It not only provides you free cloud storage, but also provides application programming interface so you can upload files via programs. This requires some minimal setup. Once set up, you can proceed to add the file upload feature to your code. I am using Python to do the job. It is quick and easy. Plus, you can port your code to any operating system, such as PC running windows, GNU/Linux, Mac OSX, or Raspberry Pi. I am assuming that you have Python 3.X.

There are two versions of Dropbox Python API, V1 and V2. V2 came out around the end of 2015 and only has minimal tutorial on Dropbox.com. Nevertheless, I will use V2. Installing the API is a snap:

On Windows:

Start a command prompt and enter:

pip install dropbox

On Raspberry Pi (Raspbian Jessie) or Debian PC

sudo pip3 install dropbox

If you still have Raspbian Wheezy, pip3 may tell you that there is a newer version of pip. Don’t attempt to upgrade your pip or pip3. It will break.

Here is a short Python script to upload and download files:

[code language=”python”]
import dropbox
file_name=’test_image.jpg’
dropbox_path=’/’
dbx=dropbox.Dropbox(‘Your access token’)
with open(file_name, ‘rb’) as f:
    dbx.files_upload(f.read(),dropbox_path+file_name,mute=True) # The change from f to f.read() was made to comply with the late-2016 change in the API.

dbx.files_download_to_file(‘Copy of ‘+file_name,dropbox_path+file_name)
[/code]

This was easy. There are only three functions that I used, first an authentication, then upload, followed by download. Your access token is generated by Dropbox. This post has the details.
The function dropbox.Dropbox() returns an object. You can use this object to upload or download files and more.
The files_upload() function does the upload. You need to first open the file on your local computer with open(). I am assuming the file is stored in the same folder as your Python script. This returns a file handler f. Then pass f to the files_upload(). This is the first argument. The next argument is the path you want the file to be uploaded to dropbox. It has to start with ‘/’, then the file name (including additional path). You don’t have to preserve file name. The third argument is useful. When you sent mute to True, you won’t get notification for the file upload. If you are making a camera trap, you don’t want your PC flooded with Dropbox notifications just because a bunny decides to visit your backyard.
The files_download_to_file requires first a local computer name, then the Dropbox file name. Again you don’t have to preserve file name. It’s useful to first check if the file exists on your local computer and decide what to do (overwrite, or add prefix/postfix to new file).
This is it! How you use it is up to you.

If you are interested, you can explore the rest of the API, such as creating folders, moving files, listing folders, etc. by reading the documentation (alert: document is very dry)

https://www.dropbox.com/developers/documentation/python

In my next post, I will make a simple Raspberry Pi camera logger to activate only during several intervals of time of the day.

Filed under Coding, Python, Raspberry pi

Connect to Dropbox

February 20, 2016 3 Comments

If you are making a data logger, or camera trap, this is definitely good news. With the right approach, you can sit in your home and data/pictures just flow to your desktop. All you need is internet connection and some Dropbox programming. With a raspberry pi or a pc, you can have your data logger or camera logger automatically upload data to your Dropbox. On a windows PC, this is nothing more than saving your data file to a folder inside your Dropbox folder. Still, constantly writing to a Drpobox file is not the best way to make use of it. You won’t be able to have a meaningful history of the file since the your data logger updates the file too many times. Also, raspberry pi doesn’t have dropbox client. So it makes sense to programmatically upload your file to Dropbox using its API or application programming interface.

I am starting a series or posts to detail how to upload files to Dropbox from within your code. This post only explains how to set things up.

Using Python, it is extremely easy to upload files to dropbox. There is only one catch, you have to set up as an app developer. Here is their logic thus how to get set up:

You have some good ideas for an app. You want to store your app’s data files on dropbox so your user can access them on any device they are using. Good idea, but how to set it up?

First you need a Dropbox account yourself. Once you’re set, log on with your web browser. You’ll see three circles on the bottom left of your window:

dropbox dev-1

Clicking the dots brings up a menu. Select Developers:

dropbox dev-2

 

Click “Create app” to create a new app. I’ve already created a couple of similar apps. It looks like I’ve been trying this camera logger idea for one too many times 😀

dropbox dev-3

Make sure that you select the right options. Select “App folder” access so all your actions are contained in a folder inside the App folder. It used to be called sandbox access. You may not want this app to access your entire Dropbox folder and accidentally delete files it is not intended to touch.

dropbox dev-4

Once done, you will be able to edit your project. Scroll down to find “Generated access token”. This will give you a very long alpha-numeric sequence that grants access to your Dropbox app folder, or the entire Dropbox if you chose “Full Dropbox”. Safeguard this string. If you need to ask questions about your code, remove/black out this string. Anyone with this string can do anything they want with your Dropbox app folder, no password needed. alternatively, you may try the App key and App secret path. That is a longer path than we need to take, just to show how things work.

dropbox dev-5

Now you are all set to develop this app, which is probably a program that uploads data/pictures to your dropbox. Read my next post about how to access Dropbox using Python. Python is one of the least preferred language I use but there are benefits of using it. It’s quick to see results. I am a big fan of Arduino. Unfortunately, it is almost impossible to upload to Dropbox using just an Arduino (except Arduino Yun).

Filed under Coding, Python, Raspberry pi

SDI-12 USB adapter on Raspberry pi

February 8, 2016 2 Comments

[UPDATE] Please follow this link to the SDI-12 USB adapter page for up-to-date information:

https://liudresllc.com/gadget/sdi-12-usb-adapter/

I have successfully developed Python code to run the SDI-12 USB adapter data logger on Raspberry PI. It was quite a learning experience for me but I was able to boil down the steps into a tutorial on how to install the latest Python on Debian/Raspbian here.

Here are some screen shots of the data logger:

2016-02-03-171037_1680x1050_scrot

2016-02-05-100346_1680x1050_scrotI was using a Decagon 5TM soil temperature and humidity sensor (hanging free in air and sometimes touching my table).

I have translated the Tera Term scripts into Python so it would run on all operating systems, including Win 10, Linux (Debian), and Raspberry PI (Raspbian). I have not got my Mac fixed but don’t expect any issues.

Now you have two Python scripts:

The configuration script detects the SDI-12 sensor’s address, prints the name of the sensor, and you can change it to a different address.

The data logger script lists all available serial ports to let you choose the correct one, then asks for sensor address, total data points, delays between data points, and also what time (GMT or local) to use. You can expand the script to take data from multiple probes and upload data to a server. I’ll add more info on the server upload to sparkfun’s phant data server.

While developing the code, I also discovered that the serial port console was very useful. You can set up auto login and run the data logger script upon auto login. This way the data logger starts logging data once the RPI boots up and auto logs on. No need to log in or use VNC (not good if your logger is using 4G hotspot for internet). Just a command prompt is all that is needed. Next step is to add a config file so the data logger will no longer ask for user inputs when this file exists and takes parameters from the file. This config file will reside in the FAT partition so that the user can easily update the parameters without having to boot into raspberry pi. Just remove the sd card and change parameters on a PC.

Filed under Announcements, Coding, Gadgets, Python, Raspberry pi

Debouncing a mechanical rotary encoder

June 21, 2014 Leave a comment

As an active member of the Arduino user community, I felt I should keep up with my library updates. So after some planning and “careful” coding, I’ve added a new digital rotary encoder class to my phi_interfaces library. The phi_interfaces library includes buttons, rotary encoders, analog buttons, joysticks, matrix keypads, serial ports, etc. as input devices and one can use getKey() to sense any of them.

http://liudresllc.com/libraries/phi_interfaces/

The original rotary encoder class already works well. This addition expands the scope, supporting both normally open and normally closed encoders, and also supporting encoders that have twice the detents as they have complete pulses (have some hardware but haven’t tested yet). Also, this addition paves the way of an analog encoder class that will help people using nano/ATMEGA328P-AU save two pins, which has two analog-only pins with no digital functions.

Along the way I met some difficulties that I thought was useful to share. Although it stems from library development, the nature of the problem is actually misunderstanding of hardware.

Hardware I used: Bourns PEC11 series 18 pulses/rev. encoder with 18 detents./rev. (no obsolete, replaced by PEC11R)

So here is what I discovered: the mechanical rotary encoder that I have has a delay between channel A closing and channel B closing about 14 ms, then delay between channel A opening and channel B opening about 8 ms. Not only is opening asymmetric with closing, but they are also faster than my expectation. I started programming the new class expecting to debounce the encoder. I first used 20ms for debouncing, like what I do for tactile switches. That didn’t yield good results. I thought I need longer time, 50 ms, not good, or about the same, only 1/3 detents got detected. I then tried shorter debouncing like 10 ms and still few detents got detected. Running the original rotary encoder code I developed initially, without debouncing, was find, with every detent registered. So I checked the encoder with a logic analyzer. Wow the action was faster than I expected, 14ms and 8 ms. I also looked at the bouncing of the contacts. It was around 0.5 ms after initial contact was made, many bouncing occurred. So if I’m using millis() to debounce, I’d rather not debounce at all.

Here is one part of the pulse when channel A closes then channel B closes, about 14 ms:

Here is another part of the pulse when channel A opens then channel B closes, about 8 ms:

Here is a typical bouncing of the contacts, in the order of 0.5 ms:

An afterthought, I read the spec sheet, which says at 15RPM, close to how fast I was turning the shaft, bouncing is 5ms max. I’ve seen bouncing around that length in time. If I read the spec sheet, I’ve probably avoided the whole learning experience smiley

Here is diagram showing symmetric pulses on the spec sheet:

So, take the spec sheet with a grain of salt.

BTW, I’ve recently designed a system that uses one analog pin to sense a mechanical rotary encoder. The purpose was to save pins. The ATMEGA328P-AU has two analog pins that have no digital functions. Why wasting them if I can hook a rotary encoder to one of them?

Filed under Arduino, Circuit basics, Coding, Sensors Tagged with , ,

How to use serial port pins as digital pins

April 2, 2014 Leave a comment

If you are using Arduino MEGA 2560, you have 4 hardware serial ports (USART port to be precise). Not everyone needs to use all of them. You can, if you want, use Serial1,2,3 as regular digital pins, or GPIO (general purpose input and output). All you have to do is to call Serial1.end() before you set RX1 and TX1 with pinMode and digitalWrite/read.

What if you are in my situation: I have a few serial port sonic rangers. They have an enable pin that enables ranging if they are held high for over 20us. It also has a free running mode, with the enable pin internally pulled up. I was using the free running mode until some interference between two rangers were discovered. Now with my shield already designed and a few dozens in service, I need a way to retrofit them to work with a screw terminal block. Arduino header and jumper is not the solution when these units are deployed in the field. Fortunately I routed both TX and RX to the board’s edge and I can solder on some 2-pin block. But I still need RX to function so I can’t call Serial1.end().

This would need some register operation. If you are interested in the whole detail, the doc2549 from atmel has everything in chapter 22 (USART). There are example codes on P211. It’s pretty clear that the Transmit only functions when the TXEN1 bit of the UCSR1B register is set. The following is from that chapter:

 

The USART Transmitter is enabled by setting the Transmit Enable (TXEN) bit in the UCSRnB

Register. When the Transmitter is enabled, the normal port operation of the TxDn pin is overridden

by the USART and given the function as the Transmitter’s serial output. The baud rate,

mode of operation and frame format must be set up once before doing any transmissions. If synchronous

operation is used, the clock on the XCKn pin will be overridden and used as

transmission clock.

 

So all we need to do to get the GPIO function back is to reset that TXEN1 bit as such: cbi(UCSR1B,TXEN1);

I tested this and it worked. I was able to drive the sonic ranger with the TX1 pin and still receive data on the RX1 pin.

Filed under Arduino, Coding

#define

January 23, 2013 Leave a comment

It may be just me but I am interested in reading code written by others. It’s like literature. Everyone has a different style and you can always pick up something useful if you read. But some beginners read code written by others simply because they wanted to use that code on their own projects. If you have already done that as a beginner, you may have seen what I am about to introduce: the #define.

These “#define” appear in the beginning of some project code and you may wonder what they do and why they are there. Here is some light reading about the “#define”:

First of all, the #define is not declaring or defining a variable. It is simply a substitution rule.

Example:

#define foo bar

If you have the above at the beginning of your code, then every time foo shows up in subsequent code, it gets replaced by bar. Normally only complete match is replaced so foobar won’t be replaced by barbar. Also foo inside a text string is not replaced either.

This “#define” command is called a preprocessor directive. The substitution occurs before the code is compiled, thus the name.
There are a lot of good use for this directive. If you want to blink an LED, you can do:

 digitalWrite(13,HIGH);
 delay(1000);
 digitalWrite(13,LOW);
 delay(1000);

But that would suck if you want to blink a different pin. You have to change pin number in multiple lines and pray not to make a mistake. But this will work much better.

#define LED 13
digitalWrite(LED,HIGH);
delay(1000);
digitalWrite(LED,LOW);
delay(1000);

This way if you move the LED to pin 10, all you have to do is to modify the “#define LED 13” into “#define LED 10”. Obviously in this example the first way only requires two changes to blink LED on pin 10. On the other hand, if you have programmed the game Angry Bird with earth gravity g=9.81m/s/s and suddenly want to expand your game to moon, where gravity g=1.62m/s/s, good luck to you if you typed in 9.81. There would be thousands of places where 9.81 was typed in and need to be changed. And from reading the program, you will have no idea what gravity value was used (just seeing a 9.81 in a calculation doesn’t count as knowing gravity is 9.81). Now if you did “#define g 9.81” in the beginning of your code, remaking your Angry Bird on the moon will only take two seconds on your coding and it will also be obvious to the reader of your code, which is probably 90% chance yourself at a later time. You will be surprised how forgetful you are.

There are a lot of advanced uses of “#define” to control how your code gets compiled, which in my opinion is not useful for beginners. I will cover that in a separate post. Just as a teaser, let’s think about this question: you have two prototypes, prototype1 with a 16X2 LCD and 4X4 matrix keypad, prototype2 with a 20X4 LCD and rotary encoder. Your project code works on both hardware but the initialization part is dependent on which prototype you are using. You will be able to use something like “#define prototype1” to make the compiler only compile initialization code for prototype1. Just exactly how, I will cover it in a separate post.

Filed under Arduino, Coding Tagged with ,

Arduino Emulation on PC

April 15, 2012 6 Comments

I have worked on this for a while and was able to run arduino code on my PC. This is to make debugging much easier and compiling much faster (1 second compared with minutes of compile and upload). Since the purpose was to develop phi_prompt programs, I only made emulations on key Arduino functions, liquidcrystal, DS1307, phi_interface, and phi_prompt libraries. There are no real hardware hooked up to arduino. Future development may include some limited feature such as digitalWrite or other hardware that I might need for projects. Here are some pictures to accompany my previous post about the professional development kit for phi_prompt:

The supported Arduino stuff is here:

millis();
microSeconds(); //Good luck with this on a multitasking operating system!
delay();

PROGMEM related definitions and functions. Seems to be working with “c-strings in PROGMEM and a pointer array in PROGMEM to point to these strings”. I have not tested other data types. Be careful with this though, pitfalls may still exist
strcpy_P();
strlen_P();
strlcpy(); // This is only available with GNU C not a standard one but I need it.
pgm_read_byte();
pgm_read_word();

LiquidCrystal library. Will work on write(char), print(int) and print(float)

Everything except for below functions, which I don’t use
        void noDisplay();                                                         ///< Turns off the display
        void display();                                                           ///< Turn on the display
        void scrollDisplayLeft(); // Hate these scroll functions. Useless
        void scrollDisplayRight();
        void leftToRight();
        void rightToLeft();
        void autoscroll();
        void noAutoscroll();
        void createChar(uint8_t, unit8_t*); // I can’t create char on PC

DS1307 all working except for this one I didn’t implement. Easy to do though. Also start and stop have no actions. You can’t change DOW without changing dates.
    int get(int, boolean);

phi_interfaces input device library only has the phi_serial_keypads class which takes inputs from keyboard. Function keys are wsadfg of course (who didn’t play some FPS games?)

phi_prompt user interface library works except that the right arrow looks different and the scroll bar is not as awesome as on real LCD

Filed under Arduino, Coding Tagged with , , ,

Updates to phi-2 shield and phi_prompt project codes

March 5, 2012 Leave a comment

I have updated the following three project codes to run on arduino 1.0 with the support of phi_interfaces and phi_prompt libraries version 1.0:

  • Interactive GPS logger
  • Alarm clock
  • Morse code translator

All of the above projects have two interfaces to operate, thanks to my most recently phi_interfaces library

  1. The local interface with buttons (I assumed the hardware to be phi-2 shield)
  2. A remote interface via serial port.

This helps you debug. Also when you need to enter text, such as in Morse translator under encode, you can simply type up a message in Arduino serial port monitor and hit send. Make sure the baud rate matches your serial port and select “no line ending” in Arduino’s serial monitor.

All of the above codes are posted on my phi_prompt google code site. Please find the actual links on the phi-2 shield page under projects. I am also interested in updating the sonic ranger project code. That will be next.

Filed under Announcements, Arduino, Coding

sscanf

January 30, 2012 4 Comments

Last time I blogged about how to use sprintf to format output, especially with numbers, into strings. This time it is sscanf, the reverse, how to extract information, such as numbers from a formatted strings.

A lot of situations call for such a process. Say you want to command your robot to move its arm by telling it to a certain angle on a certain motor. Also you may want to set the time on your arduino’s on board real time clock, or want arduino to play an MP3 clip with a sound shield out of several clips. Maybe you want to control fan speeds of your HVAC system over the internet.

Back to the robot problem, say the arm has 3 motors, rotation around a vertical axis from the base, rotation of the arm up and down, and rotation of the grabber along the arm. A natural way is to pass such a command as “MA 150 MB 30 MC 25” to command the three motors to rotate to these angles, MA to 150, MB to 30, MC to 25. Using a string, you can either type up the command in arduino IDE’s serial monitor, or have a program on your PC generate this string to control the robotic arm. The following is pseudo code to process this:

  • Define 3 variables to store extracted numbers
  • Extract each number according to a predefined format
  • A final check to make sure extraction is successful

The more actual pseudo code:

#define expected_number_of_extracted_values 3

char input_string[]=”MA 150 MB 30 MC 25″;

int A_val, B_val, C_val;

int result;

result=sscanf(input_string, format description, list of variables to store the extracted numbers);

if (result!=expected_number_of_extracted_values) complain to user.

So let’s look at the format description. This is very similar to sprintf. The expected string has a character or string followed by space then a number, then repeat two more times for 3 parameters. This is described by the following formatting string: “%s %d %s %d %s %d”. The “%s” is a space holder for a string. The MA, MB, an MC are strings. The “%d” is a space holder for an integer. So 150, 30, and 25. So it reads “string integer string integer string integer”.

Next, you decide to not care about the actual string being MA or MB or MC because you always type them in that order. We will make a less strict assumption later. This strict requirement makes the first number always for motor MA, the second number for motor MB etc. Actually you don’t even need MA MB or MC in the string if you so decide the order of the numbers. But sometimes the string is not generated by you so there are junks and we need to extract number around them.

According to what we discussed in sprintf, the format string needs to have corresponding variables, so the actual sscanf will look like:

sscanf(input_string, “%s %d %s %d %s %d”, string1, A_val, string2, B_val, string3, C_val);

Since you don’t care about extracting the strings “MA”, “MB”, and “MC”, you can use this format string:

“%*s %d %*s %d %*s %d”

Notice the stars in front of the s. Any format term with a star makes the function extract the content and then discard it. This is different from omitting “%s” in the format string and making it “%d %d %d”. If you omit the “%s” in the format string, sscanf will try to match the first thing it sees with an integer and will fail since the first thing it sees is “MA”, which is not a number. It returns failure.

So to summarize, you need the following actual command:

sscanf(input_string,”%*s %d %*s %d %*s %d”, &A_val, &B_val, &B_val);

The ampersand or ‘&’ extracts address of the variables instead of values of them. With address, sscanf can return extracted values to variables.

Actual code:

#define expected_number_of_extracted values 3

char input_string[]=”MA 150 MB 30 MC 25″;

int A_val, B_val, C_val;

int result;

result=sscanf(input_string,”%*s %d %*s %d %*s %d”, &A_val, &B_val, &C_val);

if (result!=expected_number_of_extracted_values) fail();

For more detailed information on sscanf, read this page:

http://www.cplusplus.com/reference/clibrary/cstdio/sscanf/

Filed under Arduino, Coding Tagged with

← Older posts

%d