Regex Website Parser

---

@thex said:

Best would be a community/forum-area where people can request a regex and share theirs.

Agree, that’s a good idea. You should be our master of regex then, as you are familiar with concepts like non-capturing groups

I can also recommend https://myregextester.com/index.php which also gives in indication on execution time (overhead). This may help to optimize the regex in some case.

Done

Here’s the new category Logs, Parsing, RegEx eval.
You can find as sub category in Rules and Scrips
https://forum.pimatic.org/category/30/logs-parsing-regex-eval

Can anybody explain me what i must doing to get this kind of device? Is there a json plugin, which plugin i need?

thanks

Hello Dering,

Your example with PhantomJS is not working anymore. Or my PhantomJS is not installed well. I always catch the Error:
pi@raspberrypi3:~ $ phantomjs Boersenkurs.js
TypeError: undefined is not an object (evaluating ‘document.getElementsByClassName(‘leftfloat big’)[0].innerHTML’)

undefined:2
:3
TypeError: null is not an object (evaluating ‘returnvalue.replace’)

phantomjs://code/Boersenkurs.js:13

In other application-code using the
var returnvalue = page.evaluate(function () {
return document.getElementsByClassName(‘leftfloat big’)[0].innerHTML;
});
statement, its the similar error.

I want to adapt this code to readout the wounderground-page of my personal weather station but it fails.

Do you have an idea, how to readout the value of ‘precipTotal’ from a string that i can read out via
var returnvalue=page.plainText;

the returnvalue looks like:
{“observations”:[{“stationID”:“xxxxxx”,“obsTimeUtc”:“2020-08-28T00:04:17Z”,“obsTimeLocal”:“2020-08-28 02:04:17”,“neighborhood”:“Räbke”,“softwareType”:“EasyWeatherV1.5.1”,“country”:“DE”,“solarRadiation”:0.0,“lon”:10.859,“realtimeFrequency”:null,“epoch”:1598573057,“lat”:52.191,“uv”:0.0,“winddir”:155,“humidity”:75,“qcStatus”:1,“metric”:{“temp”:15,“heatIndex”:15,“dewpt”:11,“windChill”:15,“windSpeed”:4,“windGust”:4,“pressure”:1009.31,“precipRate”:0.00,“precipTotal”:0.00,“elev”:171}}]}

Any help would be great.

regards Rainer

Hi, it looks like a json structure but the return string isn’t valid json, you need to replace the “ and ” by " (look at the details) and then use JSON.parse to get an object, like returnvalue-json = JSON.parse(returnvalue)
After that the value you want is returnvalue-json.metric.precipTotal

Danke für die Antwort,

ich habe mal einen Testcode aufgebaut, in dem der ‘returnvalue’-String bereits gesetzt ist. Aber irgendwie kann ich nicht auf ‘returnvaluejson.metric.precipTotal’ zugreifen: was mache ich falsch ?

var webPage = require(‘webpage’);
var page = webPage.create();
var url = ‘www.google.com’;

page.open(url, function(status) {

var returnvalue = ‘{“observations”:[{“stationID”:“xxxxxx”,“obsTimeUtc”:“2020-08-28T08:44:49Z”,“obsTimeLocal”:“2020-08-28 10:44:49”,“neighborhood”:“Räbke”,“softwareType”:“EasyWeatherV1.5.1”,“country”:“DE”,“solarRadiation”:664.2,“lon”:10.859,“realtimeFrequency”:null,“epoch”:1598604289,“lat”:52.191,“uv”:6.0,“winddir”:141,“humidity”:78,“qcStatus”:1,“metric”:{“temp”:19,“heatIndex”:19,“dewpt”:15,“windChill”:19,“windSpeed”:6,“windGust”:9,“pressure”:1003.01,“precipRate”:0.00,“precipTotal”:1.09,“elev”:171}}]}’;
console.log(returnvalue);

try{

	var rawValue = returnvalue.replace(/“/g, '"')
   		rawValue = rawValue.replace(/”/g, '"');
		       console.log(rawValue);
			   var returnvaluejson = JSON.parse(returnvalue);
			   console.log(returnvaluejson);
			   
			   console.log('precipTotal= '+ returnvaluejson.metric.precipTotal);
    }catch(e){
    console.error(e);
     }
phantom.exit();

});

Ergebnis:
pi@raspberrypi3:~ $ phantomjs test.js
{“observations”:[{“stationID”:“xxxxxx”,“obsTimeUtc”:“2020-08-28T08:44:49Z”,“obsTimeLocal”:“2020-08-28 10:44:49”,“neighborhood”:“Räbke”,“softwareType”:“EasyWeatherV1.5.1”,“country”:“DE”,“solarRadiation”:664.2,“lon”:10.859,“realtimeFrequency”:null,“epoch”:1598604289,“lat”:52.191,“uv”:6.0,“winddir”:141,“humidity”:78,“qcStatus”:1,“metric”:{“temp”:19,“heatIndex”:19,“dewpt”:15,“windChill”:19,“windSpeed”:6,“windGust”:9,“pressure”:1003.01,“precipRate”:0.00,“precipTotal”:1.09,“elev”:171}}]}
{“observations”:[{“stationID”:“xxxxxx”,“obsTimeUtc”:“2020-08-28T08:44:49Z”,“obsTimeLocal”:“2020-08-28 10:44:49”,“neighborhood”:“Räbke”,“softwareType”:“EasyWeatherV1.5.1”,“country”:“DE”,“solarRadiation”:664.2,“lon”:10.859,“realtimeFrequency”:null,“epoch”:1598604289,“lat”:52.191,“uv”:6.0,“winddir”:141,“humidity”:78,“qcStatus”:1,“metric”:{“temp”:19,“heatIndex”:19,“dewpt”:15,“windChill”:19,“windSpeed”:6,“windGust”:9,“pressure”:1003.01,“precipRate”:0.00,“precipTotal”:1.09,“elev”:171}}]}
[object Object]
TypeError: undefined is not an object (evaluating ‘returnvaluejson.metric.precipTotal’)

phantomjs://code/test.js:20

Irgendwie der gleiche Fehler: kein Objekt ?!

Könntest Du bitte nochmal drüber schauen ? Ich bin ein lausiger Anfänger mit JS.

I can’t see which output is coming from which console.log.
Could you add a small identifier like console.log("1: "+rawValue)
To get more debug info also change

console.log('precipTotal= '+ returnvaluejson.metric.precipTotal);

into

console.log('returnvaluejson= '+ JSON.stringify(returnvaluejson,null,2));

In the script you are parsing the orginal string and also the datastructure is with an array

so the right code should be

var rawValue = returnvalue.replace(/“/g, '"')
   		rawValue = rawValue.replace(/”/g, '"');
		       console.log(rawValue);
			   var returnvaluejson = JSON.parse(rawValue);
			   console.log(returnvaluejson);
			   
			   console.log('precipTotal= '+ returnvaluejson.observations[0].metric.precipTotal);
    }catch(e){
    console.error(e);
     }
phantom.exit();

@bertreb said in Regex Website Parser:

returnvaluejson.observations[0].metric.precipTotal

Thanks a lot, this code gave me the correct results.
For the values ‘obsTimeLocal’ and ‘humidity’ i’ve used the code >returnvaluejson.observations[0].obsTimeLocal< without ‘metric’ and it works at the first try. Seems I’ve understood a little bit with your help.

Only one question still exist: the value ‘obsTimeLocal’ is both, date and time 2020-08-28 21:51:44. Do you have an idea how to split them ?

regards Rainer

Humidity and obsTimeLocal are indeed without metric. The whole returnvalue object is:

{
  "observations": [
    {
      "stationID": "xxxxxx",
      "obsTimeUtc": "2020-08-28T00:04:17Z",
      "obsTimeLocal": "2020-08-28 02:04:17",
      "neighborhood": "Räbke",
      "softwareType": "EasyWeatherV1.5.1",
      "country": "DE",
      "solarRadiation": 0,
      "lon": 10.859,
      "realtimeFrequency": null,
      "epoch": 1598573057,
      "lat": 52.191,
      "uv": 0,
      "winddir": 155,
      "humidity": 75,
      "qcStatus": 1,
      "metric": {
        "temp": 15,
        "heatIndex": 15,
        "dewpt": 11,
        "windChill": 15,
        "windSpeed": 4,
        "windGust": 4,
        "pressure": 1009.31,
        "precipRate": 0,
        "precipTotal": 0,
        "elev": 171
      }
    }
  ]
}

if you only want the string values for date and time you can use

mydate = obsTimeLocal.split(' ')[0]
mytime = obsTimeLocal.split(' ')[1]

If you want to do datetime manipulations and formatting you need to create a Datetime object with -> datetime-object = new Date(obsTimeUtc);

But i think string splitting is what you need.