Pimatic process gets killed every night - causing bei Telegram plugin?

---

@OrTiN As a follow-up on the chat communication we had I think we need @Dan took into the issue. As far as I can see he is using a fork of “telebot” which seems to be outdated (100 commits behind parent). Maybe the fork needs to be updated to match up with the latest version of the Telegram APIs.

https://github.com/SenTzu01/pimatic-telegram/issues/30

Hi all,
have some news for this. The pimatic daemon kill is not related to telegram as seems. I uninstalled the plugin and the daemon is still killed for known reason. The daemon lives ~20-21h now. With telegram it was only ~8-10h. It seems really to be a kind of stack overflow.
Could it be an idea to delete all plugin folders and let the system recreate it, what you think? Raspian itself works fine as I can see - SMB, pureFTP… are running without any problems.

Think the developer is not more available. Is there anyone who is able to update the plugin. Would do it, but have no idea how to do .

Gents,

Glancing at the stack trace and reviewing some code, chances are the issue is not actually caused by Telegram plugin / telebot dependency, but the code is merely responding to an actual issue caused by the far end, or configuration in the network (=working as intended). EHOSTUNREACH actually means what it says: The host (telegram server side) is unreachable. It might help attempting to investigate if these IP addresses are reachable outside Pimatic (Ping is useless - see below), but from the system on which pimatic lives (e.g. terminal session). An interesting test would be to execute the following command from the linux console:
wget https://api.telegram.org/bot<API_TOKEN>/getMe (replace <API_TOKEN> with your bot token right after the word bot

Output example, where the 149.154.167.200|:443... connected and 200 OK are what you want to see. If there is an issue it will give a response code other than 200 OK. if this is combined with the line below not indicating “connected” it proves we need to investigate the issue outside pimatic. If the 443 connection is OK, but you do not get a “200 OK”, it might indicate a config issue such as the bot token not being valid (unlikely as @OrTiN indicated the systems seems to work apart from the log flooding.

Resolving api.telegram.org (api.telegram.org)... 149.154.167.200, 149.154.167.199, 149.154.167.197, ...
Connecting to api.telegram.org (api.telegram.org)|149.154.167.200|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 96 [application/json]
Saving to: ‘getMe’

Further reading
Unfortunately ping only proves there is a path between to systems and IP connectivity is possible. It does not say anything if two systems can establish a socket connection. Background: Ping uses ports 7 and 8 (Echo and Echo-reply respectively, using the ICMP protocol if I recall correctly. HTTP or HTTPS protocols actually live much higher in the network stack on top of TCP/IP. In layman’s terms: Ping confirms there is a road between two cities, but you still dont know if specific types of vehicles can travel from A to .

Will do the test this evening.

@dan said in Pimatic process gets killed every night - causing bei Telegram plugin?:

chances are the issue is not actually caused by Telegram plugin

What is the behavior if the retryTimeout is increased?

@mwittig Increasing the interval will result in less connection attempts, and therefore less connection failure logging.

Problem is solved!

After long time I found the problem of the process kills now. The reason was a simple shell script I wrote for my neighbor to get weather information I’m providing to opensensemap.org. Four values are get every minute. It seems that sometimes it happens that the scripts are not terminated and therefore several instances run until Pimatic crashes. When the internet was down Pimatic crashes always. Unfortunately Pimatic log provided nothing to that time.
Now I run each value request with a time limit of 10 seconds (timeout 10s). Normally a request should be done in ms, because its a simple curl command, but with the simple timeout it works now without any crashes since more then one week.

@dan said in Pimatic process gets killed every night - causing bei Telegram plugin?:

Increasing the interval will result in less connection attempts, and therefore less connection failure logging.

I’d also expect other effects like a smaller number of socket descriptors beings used. As said earlier short retry timeout may cause out of file descriptors errors or cause the HTTP connection agent to block.

@ortin said in Pimatic process gets killed every night - causing bei Telegram plugin?:

Unfortunately Pimatic log provided nothing to that time.

I remember we had a similar problem with scripting before. What can be done as part pimatic-shell-execute is to detect long running processes and log a warning message in case.

Nevertheless, shouldn’t known errors beeing catched and logged properly (by the plugin) instead of throwing them including the stacktrace?

EHOSTUNREACH is an error that could occur in different scenarios and should be catched.

I had a similar problem long time ago see here.
You @mwittig had the tip with the timeout. The difference was at least a log entry which helps to identify the problem.

@ortin said in Pimatic process gets killed every night - causing bei Telegram plugin?:

I had a similar problem long time ago see here.
You @mwittig had the tip with the timeout. The difference the was at least a log entry which helps to identify the problem.

Thanks for pointing me to that thread. I think I’ll add an automatical kill procedure after timeout as a feature (activated by default). Moreover, I am planning to add debug output on events triggered by child process, e.g. abnormal termination.