Announcement: ESPimaticRF coming soon

koffienl

OK, OK, ESpimatic 2.0 is still not released, shame on me!
But to ease the pain a little I will be releasing ESPimaticRF probably this week.

What the hell is ESPimaticRF With ESPimaticRF you can SEND and RECEIVE RF traffic from other locations than the Raspberry Pi itself. just stuff the Raspberry Pi somewhere in a closet (or install it on a virtual machine) and offload all the RF handling outside the raspberry pi.
Code is not finished for release yet, but will be soon.

You can place on or more ESP8266’s somewhere in your area and connect RF modules to it. RF signals received by the nodes will be send to pimatic. RF signals send by pimatic will be transferred to the nodes who will relay the signal to the local attached. No need for extra plugins or modifications to pimatic.

leader21

this is awesome!! looks very promising to me.

just a questions - how is the rf sending handled particularly? do all nodes send the same RF signal at the same time?
i am having rf shutters, so if there are multiple rf signals for the shutters, they might stop if there are some RF nodes sending cross data.

koffienl

Thanks

Not all code is finished yet, but the plan is as follow:

One program for every instance, you change the behavior in the config (act as homeduino or as node)
One homeduino , multiple nodes (currently only one node supported, it’s all hardcoded for debugging).
You can choose what to do with sending (do nothing, relay, local transmit or both) and receiving (do nothing, relay, local transmit or both) in the config per instance.
Unfortunately it is not possible to use blacklisting or whitelisting per protocol/unit/ID on the instances because that would need knowledge of all the protocols in the ESP. Every transmit should be matched to protocol/unit/ID before deciding what to do. Not only would I need to find out what the protocol is, I should also decode the pulse sequence to find the device.

leader21

ah ok i see! lots of possibilities for configuration, great!
i am looking forward to it!!

koffienl

Beta release : https://github.com/koffienl/ESPimaticRF
Only use in test environment, not stable for production yet! If you are willing to test, I assume you have some basic knowledge of pimatic, ESP and Arduino IDE.

Requirements:
2 ESP’s
1 set of RF sender/receiver
Arduino IDE 1.6.5 + ESP8266 core 2.3.0

Install guide:
Download all the files and place in correct folders (ESP8266 core can have older ArduinoJson library, make sure you don’t use that one!)
Compile and upload to ESP
After ESP starting up (can take some time to format SPIFFS), connect to wifi “ESPimaticRF” with password "espimaticrf"
Go to http://192.168.4.1
Enter your wifi credentials, wait for reboot
Go to http://ip_of_esp and upload ALL the files in the SPIFFS folder
Manually reboot ESP
Go to http://ip_of_esp and in the menu select configure
Make correct configurations and save, after save manually restart ESP
Done

Configuration explanation:
Mode : homeduino is the ESP connect to the raspberry pi, node is the ESP somewhere in your room
Recevier pin : Enter the GPIO (not the physical pin name) for the receiver (the receiver pin entry in pimatic-hommeduino plugin is ignored by the ESP!)
Transmitter pin : Enter the GPIO (not the physical pin name) for the transmitter (the transmitter pin entry in pimatic-hommeduino plugin is ignored by the ESP!)
Receive action : what should the ESP do with a received protocol (in homeduino and node mode this can only be received through RF)
Transmit action : what should the ESP do with a to be transmitted protocol (in homeduino mode this can only be received from pimatic, in node mode only through wifi)
Pimatic address: IP or hostname for the node to communicate with pimatic
Pimatic Port : Port where pimatic is running on
API key : The API key to use in the entire network. Must be configure the same in the pimatic-homeduino plugin!
Wifi SSID : SSID for the ESP to connect to
Wifi password : Password for the wifi

What is working?
ESP connected to Raspberry Pi is capable of sending and receiving RF through local RF receiver/transmitter
ESP connected to Raspberry Pi is capable of sending RF over wifi to the nodes
Pimatic is capable of receiving RF over wifi from the nodes
Standalone ESP (node) is capable of receiving RF and sending to pimatic
Standalone ESP (node) is capable of receiving RF over wifi and transmit with local RF
Untested but multiple node should work without a problem

Known issues:
Somehow, sometimes the wifi packet from homeduino to node gets lost. Not sure what 's the cause of this issue (wifi range? buggy homeduino code? buggy node code? Plain simple unreliable UDP code?)

Not working:
White/blacklisting of specific devices. there is allready some code to determine the protocol/unit/id and it is send to the nodes, but nothing is done yet with the information. Goal is to create a black- or whitelist per node.

Comments, bugs, issues? Please report in this topic

Harry van der Wolf

@koffienl said in Announcement: ESPimaticRF coming soon:

Known issues:
Somehow, sometimes the wifi packet from homeduino to node gets lost. Not sure what 's the cause of this issue (wifi range? buggy homeduino code? buggy node code? Plain simple unreliable UDP code?)

UDP (User Datagram Protocol) doesn’t have handschaking protocols and there is no guarantee the packages are delivered or even in the right order. Therefore it is also called “Unreliable” data protocol. Exactly the reason why the newer tcp/ip protocols and others were developed.
So that could very well be the reason for your issues. Not because of the unreliable UDP code (could be of course), but simply because UDP itself is unreliable.

koffienl

Yeah I know, but UDP is the fastest.
Sending the RF communication from the homeduino to the node is causing to much lag (see youtube clip in startpost), and that’s just with 1 node.
I’m looking into how to connect the nodes over TCP to the homeduino, waiting for some data, but can’t figure out how to keep that connection a live. (and from what I read, you can have a max of about 5 continuously connections?)

A other approach is just sending multiple UDP broadcasts and let the node figure out if it allready processed that specific request. Did a quick test with that but the homeduino ESP would crash when sending multiple UIDP packets within 500ms.

Not sure how to get the communication reliable and fast.

Update: Found the issue with the crashing ESP when sending multiple multicast packets.
Now testing with the following:

Generate some sort of ‘unique hash’ and add to the data to send
Send up to 4 UDP multicast packets
Node receives packets and compares the hash with the last received hash. If this is the same, ignore else process the incoming packet

SLiX

@koffienl said in Announcement: ESPimaticRF coming soon:

I’m looking into how to connect the nodes over TCP to the homeduino, waiting for some data, but can’t figure out how to keep that connection a live. (and from what I read, you can have a max of about 5 continuously connections?)

TCP Session ist kept alive, till client sends FIN flag in segment header (and so on, hope i tell no sh*t)

A other approach is just sending multiple UDP broadcasts and let the node figure out if it allready processed that specific request. Did a quick test with that but the homeduino ESP would crash when sending multiple UIDP packets within 500ms.
Update: Found the issue with the crashing ESP when sending multiple multicast packets.
Now testing with the following:

Generate some sort of ‘unique hash’ and add to the data to send

Send up to 4 UDP multicast packets

Node receives packets and compares the hash with the last received hash. If this is the same, ignore else process the incoming packet

Yes, with UDP you have to take care of successful transmission by yourself (if you need it).
Would be interesting to know, if your solution is still faster than TCP

If I have some more spare time, I will definitely try ESPimaticRF, thanks for that!!

koffienl

TCP or UDP should be roughly the same timeconsuming, but with TCP you have to establish a connection and keep it alive. UDP is just throwing out of the window and see what happens. The lack of error control and retransmission makes it faster.
Also, a UDP approach makes it a lot easier to communicate with the nodes. Just throw in a extra node and it will capture the multicast.
When going with the TCP route, I have to know either the homeduino IP or every IP of the single nodes (thought about doing a sort off UDP discovery on the nodes to let the homeduino respond so the node knows the IP of the homeduino to setup a TCP connection).

For now I have mad the UDP repeats a configurable option (default 3 repeats) with a pause of 50ms between each packet. So far I haven’t got any misses with this burst
Not sure what a good hashing method would be, for a quick ‘hash’ it took the number 10000000 and added the global millis() on the the node when ‘generating’ the hash. On the receiving (node) side I strip the first 8 characters and use that as the unique hash.
No beautiful, but for now it works

leader21

nice work ! i installed ESPimaticRF already and receiving is ok for me.
sending results in this error no matter what i do

20:17:39.631 [pimatic-homeduino] Sending Protocol: switch4 unit:23 id:14 all:false state:true Pin:7 Repeats:7
20:17:39.655 [pimatic-homeduino] data: "ERR argument_error"
20:17:39.663 [pimatic] argument_error
20:17:39.678 [pimatic] Error: argument_error
20:17:39.678 [pimatic]>    at /home/orangepi/pimatic-app/node_modules/pimatic-homeduino/node_modules/homeduino/lib/board.js:294:19
20:17:39.678 [pimatic]>    at tryCatcher (/home/orangepi/pimatic-app/node_modules/pimatic-homeduino/node_modules/homeduino/node_modules/bluebird/js/main/util.js:26:23)
20:17:39.678 [pimatic]>    at Promise._settlePromiseFromHandler (/home/orangepi/pimatic-app/node_modules/pimatic-homeduino/node_modules/homeduino/node_modules/bluebird/js/main/promise.js:503:31)
20:17:39.678 [pimatic]>    at Promise._settlePromiseAt (/home/orangepi/pimatic-app/node_modules/pimatic-homeduino/node_modules/homeduino/node_modules/bluebird/js/main/promise.js:577:18)
20:17:39.678 [pimatic]>    at Promise._settlePromises (/home/orangepi/pimatic-app/node_modules/pimatic-homeduino/node_modules/homeduino/node_modules/bluebird/js/main/promise.js:693:14)
20:17:39.678 [pimatic]>    at Async._drainQueue (/home/orangepi/pimatic-app/node_modules/pimatic-homeduino/node_modules/homeduino/node_modules/bluebird/js/main/async.js:123:16)
20:17:39.678 [pimatic]>    at Async._drainQueues (/home/orangepi/pimatic-app/node_modules/pimatic-homeduino/node_modules/homeduino/node_modules/bluebird/js/main/async.js:133:10)
20:17:39.678 [pimatic]>    at Immediate.Async.drainQueues [as _onImmediate] (/home/orangepi/pimatic-app/node_modules/pimatic-homeduino/node_modules/homeduino/node_modules/bluebird/js/main/async.js:15:14)
20:17:39.678 [pimatic]>    at processImmediate [as _immediateCallback] (timers.js:383:17)

would be glad if anybody else will try ESPimaticRF and can confirm if sending works or not.
i always get the arguement error and the above message.

koffienl

Can’t reproduce your problem, we’ll continue to investigate tommorow on the chat

17:48:00.963 [pimatic] info: Recreating "protocol test"...
17:48:29.419 [pimatic-homeduino] debug: Sending Protocol: switch4 unit:23 id:14 all:false state:false Pin:4 Repeats:7
17:48:29.489 [pimatic-homeduino] debug: data: "ACK"
17:48:29.508 [pimatic-homeduino] debug: data: "46432538{"buckets":{"0":"306","1":"957","2":"9808","3":"0","4":"0","5":"0","6":"0","7":"0"},"repeats":"7","pulse":"010101100101001010101100110010102","protocol":"switch2,switch3,switch4","unit":"14","id":"23"}"
unknown message received: 46432538{"buckets":{"0":"306","1":"957","2":"9808","3":"0","4":"0","5":"0","6":"0","7":"0"},"repeats":"7","pulse":"010101100101001010101100110010102","protocol":"switch2,switch3,switch4","unit":"14","id":"23"}
17:48:29.516 [pimatic-homeduino] debug: data: "UDP Packet 0 done"
unknown message received: UDP Packet 0 done
17:48:29.983 [pimatic-homeduino] debug: data: "UDP Packet 1 done"
unknown message received: UDP Packet 1 done
17:48:30.484 [pimatic-homeduino] debug: data: "UDP Packet 2 done"
unknown message received: UDP Packet 2 done
17:48:30.987 [pimatic-homeduino] debug: data: "UDP Packet 3 done"
unknown message received: UDP Packet 3 done

leader21

yes that’s strange, therefore it would be good to have others testing ESPimaticRF to see if only i have this issue

koffienl

If somone can confirm it’s working, please provide as much information as possible.
I have tested with the following situations:

Wemos D1 mini on Raspberry Pi
Wemos D1 mini on a virtual machine
NodeMCU on a virtual machine

mwittig

@koffienl said in Announcement: ESPimaticRF coming soon:

Sending the RF communication from the homeduino to the node is causing to much lag (see youtube clip in startpost), and that’s just with 1 node.

Make sure to set tcp_no_delay on the tcp at both ends of the tcp connection to disable the nagle algorithm. Kepp TCP connections opens if possible, as the connection establish is expensive due to the three-way handshake.

For now I have mad the UDP repeats a configurable option (default 3 repeats) with a pause of 50ms between each packet. So far I haven’t got any misses with this burst

Yes, but you need to be prepared for package loss. It’ll happen. Maybe it makes sense to have a look at existing R-UDP implementations.

koffienl

Got black and whitelisting working now. See new v0.0.2 on github

Working:

Blacklist a device
Whitelist a device
Set node in allow all / only blacklist / only whitelist through config

Not working:

Remove currently black or whitelisted devices (can be done manually through bwlist.json)

Known issues:

Only a few protocols are implemented yet. Black and whitelisting depends on the recognition of the protocol on the homeduino side.

koffienl

For Dutch readers : https://koffie.tweakblogs.net/blog/14123/espimaticrf-verstuur-rf-verkeer-over-wifi-vanuit-pimatic

leader21

had a teamviewer debug session with koffienl today, all is working PERFECTLY now
receiving is ok and sending works flawlessly and fast!

gonna test with two or more nodes now

Rene Arts

I was just discussing on koffienl’s tweakblog with him why the extra ESP is necessary. How much work is it to let the homeduino plugin publish the RF codes which it sends to the homeduino device (ESP or nano) to an MQTT topic?
The ESPs then can subscribe themselves to the topic and broadcast the RF message.
Biggest advantage would be that there would be no need for a direct physical connection between RasPi and ESP, and especially with MQTT other home automation platforms would be able to benefit as well.

leader21

sounds interesting!
it would either need a rework of the homeduino plugin or a new plugin for esp communication itself. guess imho a new plugin would be a better option since the homeduino plugin does not only provide rf communication.
but the advantage of this ESPimaticRF is that you don’t need any other plugin or install any software as mqtt and so on. so it’s easy to use and out of the box.
maybe in future the ESPimatic RF can be combined with some features known from ESPimatic itself.
Would be a swiss knife then

Rene Arts

I was just thinking, this could be done a lot easier I guess, be it temporarily or for testing purposes. Just assign the correct RF code to the on/off message of an MQTT switch which publishes to a topic where the nodes are listening. This also provides the option of sending MQTT messages from specific nodes or to all nodes (dependent on the topic).