integrity/plausible filter for incoming database/graph values

---

A integrity/plausible filter for incoming values inserted into database to filter unvalid values and prevent logging of these into database.

Example 1:
Usual temperature sensor value is around -50 to 150 but a faulty value of 4500 comes in, now the filter kicks in and disregards the value instead of saving it to database and graph.

Example 2:

filtering of values way out of line compared to other database values around that value

These filters should be able to be applied per device or per type of value.

@betz0r said in integrity/plausible filter for incoming database/graph values:

tegrity/plausible filter for incoming values inserted into database to filter unvalid values and prevent logging of these into database.
Example 1:
Usual temperature sensor value is around -50 to 1

See pimatic-filter. Additional filtering strategies can be implemented on request.
A simple outlier removal like you suggested should be easy to implement

@mwittig said in integrity/plausible filter for incoming database/graph values:

@betz0r said in integrity/plausible filter for incoming database/graph values:

tegrity/plausible filter for incoming values inserted into database to filter unvalid values and prevent logging of these into database.
Example 1:
Usual temperature sensor value is around -50 to 1

See pimatic-filter. Additional filtering strategies can be implemented on request.
A simple outlier removal like you suggested should be easy to implement

Thx mwittig for your fast response, i totally forgot about this filter plugin because i was too scared to create an extra device for every filtered value.

I will check your code and maybe we can fit a useful “everyone-can-use” logic for already saved values of a device so the filter can be applied to existing values in the database and future incoming values of this device.

@betz0r said in integrity/plausible filter for incoming database/graph values:

I will check your code and maybe we can fit a useful “everyone-can-use” logic for already saved values of a device so the filter can be applied to existing values in the database and future incoming values of this device.

I am happy to collaborate on this, i.e. if you like I can draft a new filter cutting off outliers. Possibly we can also add some additional “stats” like the number of outliers per given time interval.

@mwittig That sounds awesome. I just started digging into the plugin’s code but i never got intouch with pimatic plugin dev until now, so even as a web dev it is hard for me to immediately start with adding an outlier filter. But i realized editing database values is not part of this plugin and would probably go too far for this matter.

My 2cents on the new SimpleOutlierFilter:

• fixed range min - max, values not in that range are discarded.
• relative comparison: i.e. is this value more than x% off to the last y values? -> discard this value.
• cumulative value: this value should never decrease, if decrease, discard the last “high” value
• stats: how many faulty values do we have in given time?

In the screenshot above the value “Brennerstunden” is a cumulative value but cumulated by the central heating already. As cumulative values rise over time i think we need to handle them “extra” compared to other filter values like temperatures or humidity.