Wrong data from pimatic-openweather

---

Anyone else having trouble getting correct data from openweather?

I’m having some trouble with the pimatic-openweather plugin. Sometimes it works and sometimes it returns wrong answers. Right now it reads 14.9°C. Definitely not true (right now still ~32°C)! At least the failed data is a constant of 14.9°C, but one can only guess the right temperature from the zig-zag in the graph.

Screenshot: Black is the data from the plugin and blue is indoors for comparison.

The interval is set to 600000, which should be every 600 seconds or every ten minutes.

@Marie Same problem here. I have opened a support ticket at http://openweathermap.org/ but they don’t think it is a problem on their side. To let them reproduce the issue easily there is an action on me to write a tiny script which periodically fetches data using the curl command. This way it can be shown it is not an issue of the client library used. I’ll work on this on the weekend and will keep you posted when I have some news.

@mwittig Thank you.

@Marie As promised I have compiled the data for the openweather service desk. I’ll keep you posted as soon as I get a response.

https://gist.github.com/mwittig/5d792dc784b79803b308920dcc661589

@Marie Still no joy. The support desk at openweathermap.org is really letting me down. Since I did not get a response for weeks I created at new ticket last week, but I did not even get an entry confirmation. Maybe these guys are close to going bust, I don’t know.

To mitigate the problem I am now working on a blacklisting feature which can be used to ignore outliers. Apparently, the outliers remain constant for a long like your “14.9°C” which is “7°C” in my case (seeing this for months now). I’ll also add a feature to detect potential outliers. This should make it a little bit easier to configure the blacklisting feature.

@mwittig Thank you very much. My false data points got less, but still appear sometimes. I think they were gone, when I moved to another city in July.

I still log the old one, so I can have a look as soon as I’m back home.

Yep, wrong data points are still there.

@Marie said in Wrong data from pimatic-openweather:

Yep, wrong data points are still there.

OK, that’s bad. I am going to push my changes today which contains a new feature to blacklisting outliers. It needs to be setup manually however.

Notable Changes with v9.0

• As some users had problems with sporadic “location not found” errors, the cityId property has been added
  which allows for setting the location id to work-around the problem
• The blacklist property has been added to OpenWeatherDevice which allows for defining a list
  of weather results to ignore as they contain false data
• The day property default value of the OpenWeatherForecastDevice has been changed to 2 to obtain the forecast
  for the next day by default. The property denotes the forecast day starting from 1 to 16 where 1 means today.

@mwittig I just re-installed with git pimatic-openweather. Pimatic plugin page now show version 0.9.0.
I added the CityId (from the list provided) but still get the Error 404 City not found.
Do I need to delete the location attribute?

Thank you very much

I added the cityId for both cities I log. After pimatic restart I get one Error: City not found (404). Maybe the message could be expanded to tell me, which city could not have been found?

The blacklist feature is a very nice addition. Where do I see the strings needed to fill the base property? Right now I only blacklisted the wrong temperature (see below) which results in two error messages:

1. Invalid config of device myDeviceId: missing property "base" in /blacklist/0
2. Invalid config of device "myDeviceId": missing property "base" in /blacklist/0. Same device, just in quotation marks.

"id": "myDeviceId",
"class": "OpenWeatherDevice",
"name": "Wetter für ___",
"location": "___, Germany",
"cityId": "digits",
"units": "metric",
"lang": "de",
"timeout": 600000,
"blacklist": [
  {
    "temperature": 14.9
  }
]

The digits were taken from the url http://www.openweathermap.org/city/digits after I looked up my specific cities.

@Petjepet said in Wrong data from pimatic-openweather:

Do I need to delete the location attribute?

No. The location property must be set as it is mandatory, but it may be an emty string if the the cityId is set. Note, cityId must be a string eventhough it is a number (see example below).

{
      "location": "",
      "id": "openweather",
      "name": "openweather",
      "class": "OpenWeatherDevice",
      "timeout": 300000,
      "blacklist": [
        {
          "base": "stations",
          "temperature": 7
        }
      ],
      "cityId": "2852458"
    },

@Marie said in Wrong data from pimatic-openweather:

Where do I see the strings needed to fill the base property? Right now I only blacklisted the wrong temperature (see below) which results in two error messages

Hi. You can set "debug" to true in the plugin configuration (or use the Plugin Editor to do so). after restarting pimatic you shoild set debug message like the following:

 20:47:59.743 [pimatic-openweather] Weather data received: {"base":"stations","id":801,"temperature":16.1}

For an blacklist entry the baseproperty is mandatory. A blacklist entry may
consist of up to three of the following properties.

• base: a string indicating an internal identification for the type of weather station network. For example,
  “cmc stations” refers to the weather stations of the Canadian Meteorological Centre while “stations” appears to be
  weather stations operated on a private basis. This property is mandatory
• “id”: The weather condition id set in the blacklisted result. This property is optional. If provided, the value
  match will be “ANDed” with the remainder of the filter. For more info on weather condition ids see http://openweathermap.org/weather-conditions
• “temperature”: The temperature set in the blacklisted result. This property is optional. If provided, the value
  match will be ANDed with the remainder of the filter

@mwittig thanks for the info.
Since I still see the Error 404 City not found, is my version correct 0.9.0 (installed with Git). I believe I also saw 0.9.6 somewhere in the forum.
How to install the latest version (I’m rather new to this).

@Petjepet easiest way to update the plugins is the update section in the left menu of pimatic. Go there and pimatic will automatically search for updates of all your plugins and you can simply click on the install button. When done it will ask you to restart pimatic.

@Anduril I guess this is not the case with Openweather.
The plugin page shows v 0.9.0 is installed and the update page shows all is up to date.

@Petjepet said in Wrong data from pimatic-openweather:

I believe I also saw 0.9.6 somewhere in the forum.
How to install the latest version (I’m rather new to this).

No! You already have the latest version of pimatic-openweather which is v0.9.0.
Please send or pm me the cityId you’re using.
In case you haven’t used the device editor please also post the device configuration as a typo may be the problem.
Thanks

@mwittig See below my device configs. The City Id was added in the device definition option in pimatic. Other data was already manually edited in the config file in Pimatic 0.8 before I switched to 0.9.

{
  "id": "weather",
  "class": "OpenWeatherDevice",
  "name": "Weer nu",
  "location": "Elten, DE",
  "units": "metric",
  "lang": "nl",
  "timeout": 900000,
  "xAttributeOptions": [],
  "blacklist": [],
  "cityId": "2930665"
},
{
  "id": "forecast",
  "class": "OpenWeatherForecastDevice",
  "name": "Weersverwachting 1 dag",
  "location": "Elten, DE",
  "units": "metric",
  "lang": "nl",
  "timeout": 900000,
  "day": 2,
  "xAttributeOptions": [],
  "cityId": "2930665"
},

@Petjepet I took your config and test it for 12 hours and I also had the error “Error: Not found city (404)” once. How often does it occur on your setup? How many pimatic-openweathermap devices do you have in total? Just the two?

Apparently, it is regarded as “normal behaviour” according to the following post on a similar case which is mildly amusing. Regading the recommendations given in the referenced post I think they are more or less followed by pimatic-openweather. The only exception I see is, that requests for multiple devices (e,g. for a Weather and Forecast device) may be sent at about the same time which is a voilation to recommendation (2).

“To make it clear: a problem of our API is that errors like this don’t necessarily mean what they say. A 404 or, say, 503 error can happen even when your request is perfectly valid and when the place actually is in the database, the reason behind is the free servers overload. We do our best to extend the capacity continuously but don’t always manage fitting in the resources we have. The only thing you can do about it is following the efficient usage recommendations as this will give your requests higher priorities and let the servers deal with your requests better in general, the recommendations are: …” https://openweathermap.desk.com/customer/portal/questions/13084185-error-not-found-city

@mwittig these two are the only devices in my setup.
In de past 3 days the error did not occur. But When installing the latest version it did some other setup changes (not Openweather related) where I had to restart pimatic a few times for testing. In that phase I always had at leas one of those errors.

I guess, regarding your findings, it is to be regarded as “normal behaviour”.