[Logo] Anarduino and HopeRF Community Forum
  [Search] Search   [Recent Topics] Recent Topics   [Hottest Topics] Hottest Topics     [Groups] Back to home page 
[Register] Register / 
[Login] Login 
Over the air reprogramming and file transfer in a full network  XML
Forum Index » Anarduino
Author Message
stevech



Joined: 07/07/2014 18:07:27
Messages: 91
Offline

The concept I've been implementing is to use a standardized network to send any sort of data, one type being a file containing new program code for remotely located device that is networked as a node.
This leads the means to:
Send a program code update to an addressed "node" within a network. From a PC/Mac/Linux computer.
Send a large file that is not a programming update to any node. This might be configuration data or other app-specific data.
Conversely, receive a large data set from a network node and store it on the PC/Mac/Linux computer. This might be data acquisition logged data/GPS, etc.

The first capability will be specific to the Anarduino MiniWireless boards. These can be ordered with a variety of data radios for 433/868/915MHz and 2.4GHz bands. Some of these radios can operate in Ham bands with fewer usage and power restrictions.

The networking will use the open source RadioHead protocols. The RadioHead library is Arduino and Teensyduino compatible. The lowest level in a RadioHead stack is a driver for the radio module or a wired data link such as UART/RS232/USB. The layers on up from this are agnostic as to what transmission media (wired/wireless), much as is TCP/IP.

Each network node (e.g., a MiniWireless board) has a unique 8 bit network address of your choice, perhaps stored in EEPROM by your app.
The RadioHead protocol choices you make include a driver for the chosen radio board and your selection of a network type (topology), including:
1. Datagram packets with error detection and error correction duty falling upon the application. Much like IP's UDP. Jargon: Unreliable Datagrams.
2. Datagrams with error correction via ACK messages, with correction done automatically within the protocol stack.
3. A routed network where each node is given a static routing table of neighbor nodes. The protocol stack uses the static routes to get data from source to destination.
4. An Ad-hoc Mesh network, where nodes dynamically discover their neighbor nodes and collectively, datagrams move in hops / relays to/from the source and destination nodes. This applies mostly when there is no RF connectivity permitting one-hop (no relay) packet / datagram flows.
5. A wired network, as simple as a UART/RS232/USB link from a network node/MiniWireless board to a PC/Mac/Linux computer's serial / USB port.
6. A multi-drop wired network with several network nodes on the wired media, e.g., RS-485 or CAN
In addition to node addressing, from/to, there is a broadcast address to send the same packet to all nodes in the network.

As discussed below, the RadioHead stack for the above is the same as an application would use to exchange data with other nodes or a bridging gateway to a computer or connectivity to a LAN and onward as needed.

Here's the status
The MiniWireless board is the first target because it has a large flash memory chip in which to store received files of AVR program code and other data, as above. The flash chip differs from that of the LowPower Wireless boards. The impetus for this project was this difference. But moreover, the impetus was to use a robust protocol stack so that the application can also use the same stack for its communications needs.

The receiving nodes use a send/receive protocol to move files. This is now the RadioHead Datagram addressed packet protocol with CRC16 error checking at the link layer and at the file level.
The AVR m328P code for this is doing basic receiving now.
A new bootloader for the mega328P as used on the MiniWireless boards is running. It is compatible with AVRdude and Atmel Studio 6 (STK500) in using the Serial UART at 57600Baud.
And the new bootloader detects if a new program code file was previously downloaded and stored in the flash chip. If so, the AVR's flash is reprogrammed by the bootloader.
This new bootloader fits in 1KB and and written in C.

An application library for use by your application can download and store into the bulk flash chip, a new program code or other type of data.
The RadioHead protocols needed for the above have been integrated.

On the PC/Mac/Linux host that does file "pushes" out to network nodes...
The RadioHead stack for datagrams has been implemented in Python for PC/Mac/Linux. The Datagrams move on the serial/USB-serial link from the computer to a "gateway" MiniWireless board.
To push out a file, you install/run the Python (v2.7) program. A graphical user interface enables you to choose what file to send to which network node. And you state if the file is program code or generic data.
If the file is a normal .hex file (from Arduino or Studio 6 or Visual Micro), it is decoded and sent via the Datagram protocol from the host computer to the gateway board and then across the wired/wireless network to the chosen network node. The binary transfer of the .hex file is 1/3 the size of sending the .hex file. It also simplifies the bootloader on the AVR.

More concisely
Install new bootloader on all your boards, using some sort of ISP. Try it in normal UART bootload usage.
Choose a MiniWireless to be a gateway - between the network and the PC/Mac/Linux computer via serial.
Install the gateway's application code.
On each network node board, add the downloader library to your application code base. It will talk to the gateway to download files and store in the bulk flash chip.

This is what phase 1 of this project is to be.
The PC/Mac/Linux GUI-based Python app is working. The Python version of the RadioHead serial port Datagram is working.
The gateway application is working to download.
Code to read/write the bulk flash chip.. tested. Is in the bootloader. Same code needs to go in the downloader library for all nodes. And downloader in gateway so it can do store-and-forward).
The gateway can also bridge packets from the network to your application on the PC/Mac/Linux box - say some PC that exchanges data/commands with network nodes in one of the network topology types as above).

This has been in work for many weeks. It's not intended to be a quick hack limited to just over the air programming without a real network. Hopefully, there's a call for such as people build systems with several or many wireless/wired nodes.

EDIT:
Here is a short video demo of the GUI running in Python on a PC doing a .hex file transfer.
The Python code also has the Python versions I've done, of RH_Serial, RHGenericDriver, RHDatagram. The AVR side has RH_serial for the gateway to PC, and will bridge to RH for the wireless side.

https://vimeo.com/100167413

steve

This message was edited 4 times. Last update was at 07/07/2014 23:26:00

grantsmith



Joined: 06/07/2014 16:15:35
Messages: 4
Offline

Steve,

Wow, this would be truly useful to me. I am designing a remote water meter reading system and it would be fantastic to be able to deploy new code to the field. Will your new library be open sourced ? Any idea when it would be available ?

Thanks,
-Grant.
stevech



Joined: 07/07/2014 18:07:27
Messages: 91
Offline

I have a lot of this working.. but not done.
Am able to send from PC via gateway to a selected network node, store in SPI flash, remote reboot, the the special bootloader reprograms AVR flash.

This is the essence. Working now on ways to have several programs stored in SPI flash and select which one to program and run.
sensorhub



Joined: 06/08/2014 04:21:11
Messages: 11
Offline

Hello Steve,
nice video and very impressive demo of your software.

I'm looking forward to when you release it.

thanks for sharing
stevech



Joined: 07/07/2014 18:07:27
Messages: 91
Offline

Hello... regarding the above work I've been doing
The computer part of this was developed on Windows 7 using Python for the GUI and the RadioHead protocols coded in Python for portability among operating systems.
I just brought that up under Linux (Mint). Not hard - that was the goal! Portability.
I'd like to try it on a Mac but I don't own one.

demo video
https://vimeo.com/103387066?utm_source=email&utm_medium=clip-transcode_complete-finished-20120100&utm_campaign=7701&email_id=Y2xpcF90cmFuc2NvZGVkfGJiZDUxMjNjNmJhZjUyYjQzNjQ4MWVmMzY3MGNhM2VjMjA2fDI2ODYzODM1fDE0MDc5NzgyMTZ8NzcwMQ%3D%3D

Steve

This message was edited 1 time. Last update was at 18/08/2014 11:47:54

stevech



Joined: 07/07/2014 18:07:27
Messages: 91
Offline


Today I ran the host software on a 3rd platform: Raspberry Pi. (same arrangement as in the video, linked above).
This is the Python coded graphical user interface application plus the Python RadioHead protocol for the link from the computer to the gateway radio to the wireless network.
This makes the computer a member of the wireless network with a network address.
And the Python GUI application supports sending new program code to any network node, including the gateway, to store in SPI flash, then install it in the AVR program memory. On remotely commanded or other reboot cause, the network node that got the new code will then copy it to AVR program memory and run that new code from now on.

So now I've run this on these computers, with three network nodes and the sniffer:
  • Windows 7/64 bit

  • Linux Mint 64 bit

  • Raspberry Pi

  • All with the app and RadioHead stack in Python 2.

    network
    Two mini-wireless AVR boards as nodes and one as a sniffer, and one Teensy 3 as a node. All with RadioHead.

    The computer and the network nodes have application messaging to do transfers, pings, telemetry reporting, etc. And coexist with an application that uses the same radio and stack/protocol for its own needs.

    stevech



    Joined: 07/07/2014 18:07:27
    Messages: 91
    Offline

    New beta software for use with RadioHead

    PacketPorts IP-like ports for messaging on small-packet networks
    and
    DXRP (Data Exchange with Remote Reprogramming)


    Video demo
    http://youtu.be/mQ2CvBVZ_mU

    PDF of the "how" - not detailed in the video
    https://childresss.opendrive.com/files?90053648_bcgJI
    sensorhub



    Joined: 06/08/2014 04:21:11
    Messages: 11
    Offline

    Hello Steve,
    great demo and great progress. Can I ask when are you planning to let out a beta version? I would load it up on my config - I have 2 x miniwireless with RFM69HW and 4 x Arduino pro mini with RFM69HW (breadboard and pt to pt wiring(!) but no RTC or FLASH).


    This message was edited 1 time. Last update was at 26/08/2014 06:46:21

    jrbenito



    Joined: 28/08/2014 15:17:27
    Messages: 15
    Offline

    Hi Steve,

    Wonderful work man! Can´t wait to try it here.

    Did you release the code already? Github maybe? I can help on this if you find it useful.

    Thnaks.

    This message was edited 2 times. Last update was at 28/08/2014 15:32:32

    stevech



    Joined: 07/07/2014 18:07:27
    Messages: 91
    Offline

    Thank you.

    I wanted Rick (Mr. Anarduino) to have a go first. That began yesterday - and when has time. Hopefully a couple of days.

    The beta version is compiled for RFM69. What others should I strive for in a beta phase? I have only RFM69 mini-wireless boards.

    Steve
    jrbenito



    Joined: 28/08/2014 15:17:27
    Messages: 15
    Offline

    stevech wrote:Thank you.

    The beta version is compiled for RFM69. What others should I strive for in a beta phase? I have only RFM69 mini-wireless boards.



    IMHO, RFM69 and RFM12B maybe.

    However I also only have RFM69 therefore enough for me too.

    BR,
    Benito
    stevech



    Joined: 07/07/2014 18:07:27
    Messages: 91
    Offline

    RFM69, which RF band? 433 or 868/915?

    jrbenito



    Joined: 28/08/2014 15:17:27
    Messages: 15
    Offline

    Hi Steve,

    stevech wrote:RFM69, which RF band? 433 or 868/915?



    Now I have only 433MHz but I am planning to do some work on 915MHz too. Since I live in Brazil these two frequencies I can use here but I cannot use 868. However, we have a GSM band extension that operates with 910MHz upper limit and another extension that starts at 927MHz (not sure, need to check). I heard about some people complaining about interference due to high TX power ERBs nearby but I could not see this by myself, actually I have some Insteon devices around the house and they operate in 900MHz band, some are RF only devices; so far so good I had no noticeble interference.

    As far as my knowledge goes 868MHz is only available for Europeans and Africans (ITU region 1) and they can't use 915MHz.

    BR,
    Benito
    stevech



    Joined: 07/07/2014 18:07:27
    Messages: 91
    Offline

    In the US, unlicensed 433MHz band is supposed to have severe transmitter-on duty cycle restrictions. Due to the myriad of one-way transmissions from wireless thermometers and the like.
    But I'm a licensed Ham radio guy so I can go up a few MHz to the ham band and do what I please. Run hundreds of watts if I wanted to.

    902-928MHz is the unlicensed band in the US and Canada; maybe Mexico (do they have an FCC?) It too has duty cycle restrictions - less severe that 433MHz.

    In both unlicensed bands, with freq. hopping you can have a higher duty cycle. These radios can hop. But perhaps most radios like this have infrequent transmissions.

    868MHz in most of the EU is unlicensed but far fewer MHz than the US 900MHz band.
    And somewhere in the 900MHz band are the uplink/downlink for the EU's 2G cellular (GSM), licensed.

    315MHz in the US is unlicensed and widely used for garage door openers with rolling code encryption.

    There's a band in the US called MURS (multi-use radio service or some such). It's around 154MHz. Limited to 5KHz FM deviation, and for data only. I wish there were HopeRF or other cheap radio modules for MURS - as is, they are pricey. That band propagates better by far than 433 and up. I wonder of other countries allow 154Mhz to be used. that's just above the 2m ham band.



    This message was edited 1 time. Last update was at 30/08/2014 22:21:23

    Charly86



    Joined: 26/07/2014 06:39:50
    Messages: 11
    Offline

    Great job guys

    Take care that the great radiohead library does not support RFM12B, so may be it won't work with them (your code, may be not the wireless programming since it's not done by radiohead lib)

    As soon as the lib will support RFM12B I think I will move to it , I could port RFM12B on it but I have no time for this now.



    jrbenito



    Joined: 28/08/2014 15:17:27
    Messages: 15
    Offline

    Hi there,

    stevech wrote:
    But I'm a licensed Ham radio guy so I can go up a few MHz to the ham band and do what I please. Run hundreds of watts if I wanted to.


    Nice! I am too and can do more or less the same here.

    stevech wrote:
    902-928MHz is the unlicensed band in the US and Canada; maybe Mexico (do they have an FCC?) It too has duty cycle restrictions - less severe that 433MHz.


    Here it goes from 902 to 927 but that uplink/downlink extension band I told ends on 910MHz and starts again on 927MHz. However, this band is not allowed for Ham anymore due to 2G extension. We lost this right.

    stevech wrote:
    315MHz in the US is unlicensed and widely used for garage door openers with rolling code encryption.


    Here the garage doors use 292MHz and also 315MHz and 433,92MHz. I don´t know about restrictions to 292 and 315, need to research. Anyway, as far as I can tell 315 and 433,92 are more common to garage doors today.

    stevech wrote:
    There's a band in the US called MURS (multi-use radio service or some such). It's around 154MHz. Limited to 5KHz FM deviation, and for data only. I wish there were HopeRF or other cheap radio modules for MURS - as is, they are pricey. That band propagates better by far than 433 and up. I wonder of other countries allow 154Mhz to be used. that's just above the 2m ham band.


    I have to investigate about this. I belive 154MHz here is used by law enforcement forces but not sure.

    Meanwhile, any news from Rick?

    BR,
    Benito
    stevech



    Joined: 07/07/2014 18:07:27
    Messages: 91
    Offline

    I was out of town last week. Back at it now!

    Anarduino.com was down for a while... maintenance or delayed reboot. So Rick must be loitering around here.
    w1ll14m



    Joined: 03/10/2014 07:24:02
    Messages: 21
    Offline

    Hi Steve,

    This sounds very amazing, I was trying to mod dualoptiboot to work with the Mini-Wireless but in the process I might have killed my 2 nodes (issues with flashing a bootloader) and waiting for my next order to arrive.
    Also I would love to hear some updates about this or if there's anything I can do to help, I would love to!

    Thanks!
    stevech



    Joined: 07/07/2014 18:07:27
    Messages: 91
    Offline

    w1ll14m wrote:Hi Steve,

    This sounds very amazing, I was trying to mod dualoptiboot to work with the Mini-Wireless but in the process I might have killed my 2 nodes (issues with flashing a bootloader) and waiting for my next order to arrive.
    Also I would love to hear some updates about this or if there's anything I can do to help, I would love to!

    Thanks!
    It's all working with the Miniwireless boards and RadioHead reliable datagrams, and using the connection PORTs in DXRP (see above video demo).
    Python program for PC/Mac/Linux/RPi with GUI works well. It initiates the over the air reprogramming and exchange of other kinds of data e.g. logs.
    The code to download the new program and store it in the SPI flash chip - works - I have yet to make it a class or module or something like that so it's easy to add to a user's app.

    This message was edited 3 times. Last update was at 03/10/2014 16:24:30

    w1ll14m



    Joined: 03/10/2014 07:24:02
    Messages: 21
    Offline

    Thanks for the update, this all sounds great! I can't wait to get my hands on this.

    This message was edited 1 time. Last update was at 21/10/2014 03:51:35

    stevech



    Joined: 07/07/2014 18:07:27
    Messages: 91
    Offline


    the latests major addition is
    Use the GUI on the PC to "ping all" in-range nodes.
    They all reply in a time-orderly manner (rather than just mash-up collisions)
    Now we have a list of radio/MCU nodes out there, the RSSI of each, the code version of each, etc.

    Then you can click once to send new code to them all (assuming that's the right thing to do). Each has saved it's unique config in EEPROM.
    The PC/Mac/Linux Python program then pushes the new code to each node in succession, with extensive error checking. Each node saves the downloaded code to SPI flash storage as in the MiniWireless boards.
    The user then clicks "reboot all", and a command is sent over the air.
    All reboot and the bootloader (mine) uses the SPI flash storage to rewrite AVR program memory with the new code. Then boot to run that.

    Now the user on the computer clicks "ping all" and see all did come up OK and all have the correct code version.

    Over the air reprogramming with automated gang programming, I guess you'd say.

    I should make a demo video. But I'm such a dweeb at videos.
    w1ll14m



    Joined: 03/10/2014 07:24:02
    Messages: 21
    Offline

    I'm intrigued! Any eta for the code?
    stevech



    Joined: 07/07/2014 18:07:27
    Messages: 91
    Offline

    I don't now know of a way to control non-commercial use of such code - as there are commercial uses now.

    suggestions welcomed.
    jrbenito



    Joined: 28/08/2014 15:17:27
    Messages: 15
    Offline

    stevech wrote:I don't now know of a way to control non-commercial use of such code - as there are commercial uses now.

    suggestions welcomed.


    By control you mean avoid the use or forbid it?

    To forbid commercial use is a matter of license but to avoid someone from silent ignore your license is another matter.

    Anyway I truly believe in opensource and if done right I am sure you would benefit of the community around your code. This, of course, does not interfere with your right of charge by your product (hardware and software).

    Regards,
    Benito.
    stevech



    Joined: 07/07/2014 18:07:27
    Messages: 91
    Offline

    jrbenito wrote:
    stevech wrote:I don't now know of a way to control non-commercial use of such code - as there are commercial uses now.

    suggestions welcomed.


    By control you mean avoid the use or forbid it?

    Regards,
    Benito.

    Control the use of code for commercial purposes - because in much of the world today, licensing agreements are ignored.
     
    Forum Index » Anarduino
    Go to:   
    Powered by JForum 2.1.9 © JForum Team