Digital Filter for Devices/Variables

Heizelmann

Does pimatic have an expression or feature for digital filtering or smoothing of incoming sensor values?
If not, this will be a new feature request for that.
If whatever incoming sensor data is noisy or have some jitter and you are not satisfied with it, then a digital filter you may add to the varaible/device to smooth the data would be nice. This might be done by math operation in a value expression (needs a data point history) or some additional attributes for filter type and parameters.

mwittig

@Heizelmann As part of the VariablesDevice you can use some functions, for example:

roundToNearest($weather.temperature,0.5) - This will round temperature values in 0.5 steps
round($weather.temperature) - just the normal Math.round function as provided by JS
min(A, B)
max(A, B)
… other for time and number formatting. See https://github.com/pimatic/pimatic/blob/master/lib/variables.coffee#L157-L266

{
      "id": "variable_device_temp3",
      "name": "TemperaturImFlur",
      "class": "VariablesDevice",
      "variables": [
        {
          "name": "myVariable",
          "expression": "roundToNearest($weather.temperature,0.5)",
          "type": "number",
          "unit": "°C"
        }
      ]
  }

Guest

@mwittig
at the moment i do this with an external script for my ultrasonic sensor:
5 measurements, kick lowest and highest value. and calculate the average from the remaining 3 values.

Heizelmann

@mwittig Thanks for the information in your first post about the existing math routines.
What I am realy looking for is what you mentiones in your last post. Something like
digitalSmooth($weather.temperature,5,1) (5 is the number of measurements and 1 the cut off on both sides of the sorted measurements array.

Concerning your last post, I guess you have a rule for executing the shell script. Can you show the script. I am interesting in from where you get the data point history. Do you take into account if values dont arrive at equal time intervals?

I like external script but on a second level or for integration of complex sub systems, so I still vote for adding this feature to Pimatic.

mwittig

@Heizelmann The last post you refrred to was had been sent by @dk00000, btw. May be @dk00000 is ready to share the script with you.

I agree, it would be nice to have a built-in filtering feature, but I am not sure how this feature should be realized. May be you write up in some more detail how the feature should look like - ideally, as precise as possible

Heizelmann

@mwittig I am not so familar with the Pimatic software to tell you how the feature should be realized.

In general what I would like to have is a digital smoothing filter for sensor values similar to what @dk00000 decribed above.

Maybe the following practical examples from the ‘arduino sensor world’ are helpfull

Yves911

Could be a nice feature

Guest

@mwittig said:
May be @dk00000 is ready to share the script with you.

its nots rocket science

this is the part of the mentioned python script:

def volumenberechnung():
  abstand1 = distanz()
  time.sleep(1)
  abstand2 = distanz()
  time.sleep(1)
  abstand3 = distanz()
  time.sleep(1)
  abstand4 = distanz()
  time.sleep(1)
  abstand5 = distanz()
  mylist = [abstand1, abstand2, abstand3, abstand4, abstand5]
  mylist.remove(max(mylist))
  mylist.remove(min(mylist))
  abstand1=mylist[0]
  abstand2=mylist[1]
  abstand3=mylist[2]
  abstand = (abstand1 + abstand2 + abstand3)/3
  wasserstand = gesamthoehe - abstand
  volumen = pi*radius*radius*wasserstand/1000
  return volumen

distanz() measures the distance from ultra sonic sensor hc-sr04

Heizelmann

@dk00000 Thanks for sharing! But, this is not excactly what I am looking for and what is shown in the links I posted above.

See the algorithms in the links I posted above. For such smoothing filters it is necessary to retreive the last N measuring points and advance only one step.

The general algorithm is y[t] = f(x[t], x[t-1], … x[t-N-1]) by sampling x at equal intervals.

Heizelmann

I am still voting for some digitalSmooth functions like digitalSmooth($weather.temperature,5,1)

Guest

@Heizelmann
aah, okay. now i understand what you mean.
than i would like to add also an improvement request for a related “problem”.
in the pimatic chart i can see the value of my roller shutters.
example: if the shutter is at 0 (point in time: t1) and after one hour (t2) i change it to 100 then the chart shows an increasing line from t1 to t2. thats wrong. because the shutters were at 0 all the time and changes just at the moment t2 to 100. do you know what i mean?

Heizelmann

@dk00000 A bit out of topic.
May be “discrete”: true helps and/or update pimatic for latest issue 721.

Guest

@Heizelmann thanks. will try it!

mwittig

@Heizelmann said:

“discrete”: true helps and/or update

I think this feature is not well-known and hardly documented. To shed some light on this it may be useful to read the following thread - http://forum.pimatic.org/topic/18/implemented-logging-and-graphing-the-state-of-devices

As far as I know the feature is currently only supported by the VariablesDevice (numbers) and the ShellSensor.

mwittig

@Heizelmann @Yves911 @dk00000

Following the discussion on digital filter I have drafted a plugin which currently provides two simple filter types. For details see pimatic-filter. Please let me know what you think.

Heizelmann

@mwittig : Might be an solution. In my opinion, a math expression applied directly to the original device or variable as I proposed above would be better than an additional device.

beikeland

I just briefly looked at using pimatic-filter for my DHT11 sensors which only output integers, so it looks a bit choppy.

But got this in my log, not sure i made the device correctly.

11:23:30.583 [pimatic] error: A uncaught exception occured: TypeError: Cannot read property 'value' of undefined 11:23:30.583 [pimatic] error:> at SimpleMovingAverageFilter.Device.toJson (/home/pi/pimatic-app/node_modules/pimatic/lib/devices.coffee:173:30) 11:23:30.583 [pimatic] error:> at Framework.<anonymous> (/home/pi/pimatic-app/node_modules/pimatic/lib/framework.coffee:492:37) 11:23:30.583 [pimatic] error:> at Namespace.<anonymous> (/home/pi/pimatic-app/node_modules/pimatic/lib/framework.coffee:492:35) 11:23:30.583 [pimatic] error:> at Namespace.EventEmitter.emit (events.js:95:17) 11:23:30.583 [pimatic] error:> at Namespace.emit (/home/pi/pimatic-app/node_modules/pimatic/node_modules/socket.io/lib/namespace.js:205:10) 11:23:30.583 [pimatic] error:> at /home/pi/pimatic-app/node_modules/pimatic/node_modules/socket.io/lib/namespace.js:172:14 11:23:30.583 [pimatic] error:> at process._tickCallback (node.js:415:13) 11:23:30.583 [pimatic] error:> This is most probably a bug in pimatic or in a module, please report it! 11:23:30.597 [pimatic] warn: Keeping pimatic alive, but could be in an undefined state, please restart pimatic so soon as possible!

I take it the expression bit it supposed to match the input variable? And this creates a new variable $filter1.temperature?

"devices"
{
  "id": "homeduino-temperature",
  "name": "Office Temp/DHT",
  "class": "HomeduinoDHTSensor",
  "type": 11,
  "pin": 13,
  "interval": 300000
},
{
  "class": "SimpleMovingAverageFilter",
  "id": "filter1",
  "name": "Filter1",
  "size": 5,
  "output": {
    "name": "temperature",
    "label": "Temperature",
    "expression": "$homeduino-temperature.temperature",
    "acronym": "T",
    "type": "number",
    "unit": "°C"
  }
},

mwittig

@beikeland Unfortunately, I am not able to reproduce the error, but from analyzing the stack trace, I think it is a race condition. It may occur when the expression is evaluated for the first time while the device has not been fully initialized. Thus, I have changed the initialization to make sure the device is fully initializated before the evaluation takes place.

I have published a new release of the package.

beikeland

I’ll report back when i get a chance to test it. Holiday mode now

beikeland

Works like a charm now! Thanks!!
Not goin to bother buying more expensive sensors now
sample graph