Regex Website Parser

---

Thinking for some time now of a simple plugin for grabbing website data.

Something like

{
  "id": "pimatic-npm-version",
  "name": "NPM verison Pimatic",
  "class": "RegexWebsiteParser",
  "url": "https://www.npmjs.com/package/pimatic",
  "regex": ".*<strong>(.*)<\/strong>\n.*is the latest"
  "type": "string"
},

https://regex101.com/r/yY2fQ8/1

should be quite easy to write…

Guess a regular expression will not always help (e.g. for dynamic loaded content etc.).

So another (universal) approach might be:
https://forum.pimatic.org/topic/371/retrieve-data-from-websites-universal-approach

A (german) introduction to PhantomJS can be found here: http://www.heise.de/ct/ausgabe/2014-26-Website-Automatisierung-mit-PhantomJS-und-CasperJS-2464003.html

hmm but this would add quite some overhead to the plugin, maybe this would be an additional plugin

@thex I think the regex approach may suffice in some cases. However, in many cases you’d need a pipeline of tools to process the web page. For example, in some cases it may be easier to convert the web page to text first. Moreover, you may need awk in some cases to fetch a column, e.g. html2text --> grep a line --> awk to fetch a column, see https://forum.pimatic.org/topic/1627/solved-ground-river-or-seawater-level-info-in-a-device, for example. Note, there is at least a nodejs wrapper for awk.

Sure but actually I don’t know why this bloated “toolchain” is used other than it might be a bit easier to configure than a regex.

My regex knowledge is also very basic but I was always able to figure out a regex which would give the value I want from the html source in < 10 min with the help of regex101.com

@thex said:

I don’t know why this bloated “toolchain” is used other than it might be a bit easier to configure than a regex

Simple answer: It is not always possible to fetch a datum from a web page with a single regex! Also keep in mind that the average user may not be able to write solid regex.

Things to look at:

• https://github.com/danielstjules/pjs
• https://github.com/agordon/webawk
• https://www.npmjs.com/package/json-query
• https://www.npmjs.com/package/json-easy-filter
• http://stringjs.com/

Sure we would need some kind of tool which will figure out the regex, I agree that this not the easiest way but very straight forward and flexible.

Regex are extremely powerfull, it should be possible to get just about any value parsed out of a html/textfile.
example for the water-level in the referenced post:

https://regex101.com/r/eO8wL7/2

Best would be a community/forum-area where people can request a regex and share theirs. Someone who is better in regex than me might figure something like this out in 2 minutes straight.

@thex said:

Best would be a community/forum-area where people can request a regex and share theirs.

That’s a good idea. We should really do this since there are many requests on how to fetch data and how to parse log files. A category for log reader and shellsensor Q&A
will have a look at that l8r.

sure, good idea, regex support also would be a great addition to the logreader

I also thought about something like jsonquery might be a second device class in the plugin where you instead of a regex supply such a json query and a url which returns json.

#Website data from json-query example
var data = {
  people: [
    {name: 'Matt', country: 'NZ'},
    {name: 'Pete', country: 'AU'},
    {name: 'Mikey', country: 'NZ'}
  ]
}


#device
{
  “id”: “newzealand-name”,
  “name”: “Name of the guy from NZ”,
  “class”: “JsonParser”,
  “url”: "https://www.peaople.com/names",
  “jsonQuery”: “people[country=NZ].name”
  “type”: “string”
},

@thex said:

Best would be a community/forum-area where people can request a regex and share theirs.

Agree, that’s a good idea. You should be our master of regex then, as you are familiar with concepts like non-capturing groups

I can also recommend https://myregextester.com/index.php which also gives in indication on execution time (overhead). This may help to optimize the regex in some case.

Done

Here’s the new category Logs, Parsing, RegEx eval.
You can find as sub category in Rules and Scrips
https://forum.pimatic.org/category/30/logs-parsing-regex-eval

Can anybody explain me what i must doing to get this kind of device? Is there a json plugin, which plugin i need?

thanks

Hello Dering,

Your example with PhantomJS is not working anymore. Or my PhantomJS is not installed well. I always catch the Error:
pi@raspberrypi3:~ $ phantomjs Boersenkurs.js
TypeError: undefined is not an object (evaluating ‘document.getElementsByClassName(‘leftfloat big’)[0].innerHTML’)

undefined:2
:3
TypeError: null is not an object (evaluating ‘returnvalue.replace’)

phantomjs://code/Boersenkurs.js:13

In other application-code using the
var returnvalue = page.evaluate(function () {
return document.getElementsByClassName(‘leftfloat big’)[0].innerHTML;
});
statement, its the similar error.

I want to adapt this code to readout the wounderground-page of my personal weather station but it fails.

Do you have an idea, how to readout the value of ‘precipTotal’ from a string that i can read out via
var returnvalue=page.plainText;

the returnvalue looks like:
{“observations”:[{“stationID”:“xxxxxx”,“obsTimeUtc”:“2020-08-28T00:04:17Z”,“obsTimeLocal”:“2020-08-28 02:04:17”,“neighborhood”:“Räbke”,“softwareType”:“EasyWeatherV1.5.1”,“country”:“DE”,“solarRadiation”:0.0,“lon”:10.859,“realtimeFrequency”:null,“epoch”:1598573057,“lat”:52.191,“uv”:0.0,“winddir”:155,“humidity”:75,“qcStatus”:1,“metric”:{“temp”:15,“heatIndex”:15,“dewpt”:11,“windChill”:15,“windSpeed”:4,“windGust”:4,“pressure”:1009.31,“precipRate”:0.00,“precipTotal”:0.00,“elev”:171}}]}

Any help would be great.

regards Rainer

Hi, it looks like a json structure but the return string isn’t valid json, you need to replace the “ and ” by " (look at the details) and then use JSON.parse to get an object, like returnvalue-json = JSON.parse(returnvalue)
After that the value you want is returnvalue-json.metric.precipTotal

Danke für die Antwort,

ich habe mal einen Testcode aufgebaut, in dem der ‘returnvalue’-String bereits gesetzt ist. Aber irgendwie kann ich nicht auf ‘returnvaluejson.metric.precipTotal’ zugreifen: was mache ich falsch ?

var webPage = require(‘webpage’);
var page = webPage.create();
var url = ‘www.google.com’;

page.open(url, function(status) {

var returnvalue = ‘{“observations”:[{“stationID”:“xxxxxx”,“obsTimeUtc”:“2020-08-28T08:44:49Z”,“obsTimeLocal”:“2020-08-28 10:44:49”,“neighborhood”:“Räbke”,“softwareType”:“EasyWeatherV1.5.1”,“country”:“DE”,“solarRadiation”:664.2,“lon”:10.859,“realtimeFrequency”:null,“epoch”:1598604289,“lat”:52.191,“uv”:6.0,“winddir”:141,“humidity”:78,“qcStatus”:1,“metric”:{“temp”:19,“heatIndex”:19,“dewpt”:15,“windChill”:19,“windSpeed”:6,“windGust”:9,“pressure”:1003.01,“precipRate”:0.00,“precipTotal”:1.09,“elev”:171}}]}’;
console.log(returnvalue);

try{

	var rawValue = returnvalue.replace(/“/g, '"')
   		rawValue = rawValue.replace(/”/g, '"');
		       console.log(rawValue);
			   var returnvaluejson = JSON.parse(returnvalue);
			   console.log(returnvaluejson);
			   
			   console.log('precipTotal= '+ returnvaluejson.metric.precipTotal);
    }catch(e){
    console.error(e);
     }
phantom.exit();

});

Ergebnis:
pi@raspberrypi3:~ $ phantomjs test.js
{“observations”:[{“stationID”:“xxxxxx”,“obsTimeUtc”:“2020-08-28T08:44:49Z”,“obsTimeLocal”:“2020-08-28 10:44:49”,“neighborhood”:“Räbke”,“softwareType”:“EasyWeatherV1.5.1”,“country”:“DE”,“solarRadiation”:664.2,“lon”:10.859,“realtimeFrequency”:null,“epoch”:1598604289,“lat”:52.191,“uv”:6.0,“winddir”:141,“humidity”:78,“qcStatus”:1,“metric”:{“temp”:19,“heatIndex”:19,“dewpt”:15,“windChill”:19,“windSpeed”:6,“windGust”:9,“pressure”:1003.01,“precipRate”:0.00,“precipTotal”:1.09,“elev”:171}}]}
{“observations”:[{“stationID”:“xxxxxx”,“obsTimeUtc”:“2020-08-28T08:44:49Z”,“obsTimeLocal”:“2020-08-28 10:44:49”,“neighborhood”:“Räbke”,“softwareType”:“EasyWeatherV1.5.1”,“country”:“DE”,“solarRadiation”:664.2,“lon”:10.859,“realtimeFrequency”:null,“epoch”:1598604289,“lat”:52.191,“uv”:6.0,“winddir”:141,“humidity”:78,“qcStatus”:1,“metric”:{“temp”:19,“heatIndex”:19,“dewpt”:15,“windChill”:19,“windSpeed”:6,“windGust”:9,“pressure”:1003.01,“precipRate”:0.00,“precipTotal”:1.09,“elev”:171}}]}
[object Object]
TypeError: undefined is not an object (evaluating ‘returnvaluejson.metric.precipTotal’)

phantomjs://code/test.js:20

Irgendwie der gleiche Fehler: kein Objekt ?!

Könntest Du bitte nochmal drüber schauen ? Ich bin ein lausiger Anfänger mit JS.

I can’t see which output is coming from which console.log.
Could you add a small identifier like console.log("1: "+rawValue)
To get more debug info also change

console.log('precipTotal= '+ returnvaluejson.metric.precipTotal);

into

console.log('returnvaluejson= '+ JSON.stringify(returnvaluejson,null,2));

In the script you are parsing the orginal string and also the datastructure is with an array

so the right code should be

var rawValue = returnvalue.replace(/“/g, '"')
   		rawValue = rawValue.replace(/”/g, '"');
		       console.log(rawValue);
			   var returnvaluejson = JSON.parse(rawValue);
			   console.log(returnvaluejson);
			   
			   console.log('precipTotal= '+ returnvaluejson.observations[0].metric.precipTotal);
    }catch(e){
    console.error(e);
     }
phantom.exit();

@bertreb said in Regex Website Parser:

returnvaluejson.observations[0].metric.precipTotal

Thanks a lot, this code gave me the correct results.
For the values ‘obsTimeLocal’ and ‘humidity’ i’ve used the code >returnvaluejson.observations[0].obsTimeLocal< without ‘metric’ and it works at the first try. Seems I’ve understood a little bit with your help.

Only one question still exist: the value ‘obsTimeLocal’ is both, date and time 2020-08-28 21:51:44. Do you have an idea how to split them ?

regards Rainer

Humidity and obsTimeLocal are indeed without metric. The whole returnvalue object is:

{
  "observations": [
    {
      "stationID": "xxxxxx",
      "obsTimeUtc": "2020-08-28T00:04:17Z",
      "obsTimeLocal": "2020-08-28 02:04:17",
      "neighborhood": "Räbke",
      "softwareType": "EasyWeatherV1.5.1",
      "country": "DE",
      "solarRadiation": 0,
      "lon": 10.859,
      "realtimeFrequency": null,
      "epoch": 1598573057,
      "lat": 52.191,
      "uv": 0,
      "winddir": 155,
      "humidity": 75,
      "qcStatus": 1,
      "metric": {
        "temp": 15,
        "heatIndex": 15,
        "dewpt": 11,
        "windChill": 15,
        "windSpeed": 4,
        "windGust": 4,
        "pressure": 1009.31,
        "precipRate": 0,
        "precipTotal": 0,
        "elev": 171
      }
    }
  ]
}

if you only want the string values for date and time you can use

mydate = obsTimeLocal.split(' ')[0]
mytime = obsTimeLocal.split(' ')[1]

If you want to do datetime manipulations and formatting you need to create a Datetime object with -> datetime-object = new Date(obsTimeUtc);

But i think string splitting is what you need.