[solved] Ground, river or seawater level info in a device

---

How to get the Dutch actual groundwater, riverwater or seawater level info into a Pimatic device.

Information:

• Measurement Map (to locate useful/local information)
  http://www.rijkswaterstaat.nl/kaarten/waterstand-tov-nap.aspx
• HTML page (used in this case to grep information)
  http://www.rijkswaterstaat.nl/apps/geoservices/rwsnl/awd.php?mode=html
• JSON Data (not used in this case. additional)
  http://www.rijkswaterstaat.nl/apps/geoservices/rwsnl/awd.php?mode=data&projecttype=waterstanden&category=1&loc=HAGO&net=LMW

Step 1 - Install html2text with:

 sudo apt-get install html2text

Step 2 - Used a ShellSensor device to grep the actual water level for example:

  {
    "id": "water-level-hagesteinbeneden",
    "name": "Hagestein Beneden",
    "class": "ShellSensor",
    "attributeName": "cm",
    "attributeType": "number",
    "attributeUnit": "cm",
    "command": "curl -s http://www.rijkswaterstaat.nl/apps/geoservices/rwsnl/awd.php?mode=html | html2text | grep Hagestein_beneden |awk '{print $4}'",
    "interval": 500000
  },

Results:

@Pimagical

try to play with this workaround

https://forum.pimatic.org/topic/289/need-help-to-read-value-from-web

@Pimagical As the service returns complex JSON data including history data it is bit tricky to filter out the right value from the dataset using grep, awk, and friends, but, of course, it is doable.

Another approach is to use a simple Node-JS program which can be executed by the ShellSensor. The following example is self-contained, i.e. no packages are required as the ‘http’ module is already part of the node runtime. To use it for the desired location (river/sea measurement point) simply select the location from the web view. Then copy the url and paste it into the script below and change mode=html with mode=data. Give it a try by executing node waterlevel (assuming you have stored the script to a file watherlevel.js).

N.B. The script is not perfect w.r.t. error handling. So, there is room for improvements.

var url = 'http://www.rijkswaterstaat.nl/apps/geoservices/rwsnl/awd.php?mode=data&projecttype=waterstanden&category=1&loc=AMRO&net=LMW';
var http = require('http');

var request = http.get(url, function(res){
    var body = '';

    res.on('data', function(chunk){
        body += chunk;
    });

    res.on('end', function(){
        var jsonResponse = JSON.parse(body);
        try {
            if (jsonResponse.hasOwnProperty('H10')) {
                var records = jsonResponse.H10;
                var lastRecord = records[records.length -1];
                // write value to standout
                console.log(lastRecord.waarde);
            }
            else {
                throw new Error("Unexpected json response data");
            }
        }
        catch (e) {
            request.emit('error', e);
        }
    });
}).on('error', function(e){
    // write to standard error
    console.error("Got an error: ", e);
    process.exit(1);
});

Very quick and dirty.
Install html2text: sudo apt-get install html2text
Use the command (in pimatic, hence the escape \ ):
curl -s http://www.rijkswaterstaat.nl/apps/geoservices/rwsnl/awd.php?mode=html | html2text | grep \"Hagestein_beneden\"

On a command prompt without the escape back slashes :

curl -s http://www.rijkswaterstaat.nl/apps/geoservices/rwsnl/awd.php?mode=html | html2text | grep "Hagestein_beneden"
Hagestein_beneden   04   Waterhoogte 72         cm NAP  LMW         HAGO        H10           meer

If you also want to get rid of the underscore use:

curl -s http://www.rijkswaterstaat.nl/apps/geoservices/rwsnl/awd.php?mode=html | html2text | grep "Hagestein_beneden" |sed 's/_/ /'
Hagestein beneden   04   Waterhoogte 72         cm NAP  LMW         HAGO        H10           meer

Edit:
So your shellsensor would contain the command:

"command": "curl -s http://www.rijkswaterstaat.nl/apps/geoservices/rwsnl/awd.php?mode=html | html2text | grep \"Hagestein_beneden\" |sed 's/_/ /'",

@Harry-van-der-Wolf @Harry-van-der-Wolf The ShellSensor reads a single value. So, you need another filter command, e.g. using awk to filter out the value from the appropriate column.

EDIT: the following should work;

"command": "curl -s http://www.rijkswaterstaat.nl/apps/geoservices/rwsnl/awd.php?mode=html | html2text | grep \"Hagestein_beneden\" | awk '{print $4}",

@mwittig
I don’t understand. My command returns a string. That can be put into a ShellSensor. I’m using that more often.
I just added my exact command to a shellsensor in pimatic and I get exactly what I want.

EDIT: I “abused” one of my existing strings for it so don’t mind the prefix “CV Control Espimatic”.

If you want to save all values to the database it would be better to indeed strip the values from the string and assign them to varibles like you are doing ith the awk command.

@Harry-van-der-Wolf Well, my assumption is that @Pimagical wants to track the water level values over time (e.g. using the graph-tool) and wants to use water level values as part of rules. If it is just about displaying it, your solution will do.

Another approach might be:
https://forum.pimatic.org/topic/371/retrieve-data-from-websites-universal-approach

@DerIng said:

Another approach might be …

Thanks for pointing to this one! I remember the German c’t magazine had a fairly detailed article on the matter. I think it is the following article: http://www.heise.de/ct/ausgabe/2014-26-Website-Automatisierung-mit-PhantomJS-und-CasperJS-2464003.html

@mwittig @Harry-van-der-Wolf Thank you guys!

I made a working html2text command from the number in cm in a ShellSensor with use of the graph-tool (use “attributeType”: “number” instead of “string”)

{
  "id": "water-level-hagesteinbeneden",
  "name": "Hagestein Beneden",
  "class": "ShellSensor",
  "attributeName": "cm",
  "attributeType": "number",
  "attributeUnit": "cm",
  "command": "curl -s http://www.rijkswaterstaat.nl/apps/geoservices/rwsnl/awd.php?mode=html | html2text | grep Hagestein_beneden |awk '{print $4}'",
  "interval": 500000
},

I don’t know if water level data is publicly available in other countries. In case of a working solution i will update this topic for that specific country.