Archived Forum

about the wifi problem this did the trick for me: http://harizanov.com/2013/04/auto-reconnect-wifi-on-raspberry-pi/

kWh feed: I am guessing that it went from a positive value to a negative in one big step, so my first thought is a number - maybe an intermediate step in the process - has overflowed and become negative. What was the last positive value and what was the first negative one? What processing is done on that to get to the feed?

yep.. from one positive to a BIG negative in one step

last value is 80.8 recorded at 14:13:40 and then no values after that.. not even 0... i then slowly go forward and i do not see a datapoint before 14:15:49... the value was now -175477.7

what i do is take the watt number transmitted and log that... i then do the kwh processing on the wattnumber and also log that...

now the same emontx also transmit my hotwater cylinder temp and numbers from my heat meter... the hotwater messurements are also completely gone in that timetimespan...but they return normal as they are direct from input to log without any processing,...

i also checked the watt feed that the kwh builds on and that one is not transmitted either in that timespan...

so i can see the emontx did not transmit in that time... but should that not have caused just 0's ?? 

my water emontx that only counts pulses and calc flowrate came through fine in the timeperiod... and they are feed from the same power supply... i feed one with and mini usb and then via 2 short wires i feed the other...

first datapoint was recored 15/10-14 14:37:30

EDIT: both sketches are attached.... just if there was a fault here

EDIT2: http://energymonitor-dk.dns4e.net/emoncms/vis/auto?feedid=78 shows the feed...

I think this is one for Trystan. I've just taken the step in Watts and divided it by the time difference in hours and it comes close to 2^32 (close enough to raise a suspicion but not close enough to be totally convincing).

oki, will keep an eye out... 

but would missing transmission cause this or would the kwh number just stay still until transmission gets back?

It's not the register value out of the Kamstrup meter then, but the power value in Watts that's being accumulated and turned into kWh in emonCMS that's the problem.

I have not got to grips with the new feed engines, so I can't give you a definitive answer on that. The sensible things to do would be either to assume nothing and interpolate between the last value and the next, or to assume zero, or to assume the last value for all the missing points. None of those would give the answer that you got, so I think the likely explanation is it saw a corrupted number and believed it.

Although boelle hasn't shown his processing of that input, the input.png shows no additional processing icons, just "log" and "kwh" so I suspect the power value is just logged to a feed and then logged as kwh's. which you would expect to be simple enough to spot any anomaly in the power feed that could lead to such a deflection.

Unless the time suddenly jumped back to 1970 for a while. ~570w for minus 35years is ~ -175000Kwh and wouldn't be noticeable in the power feed as the timestamp was 1970, there would just be a hole in the data, which he says he has.

Is it possible a read-only time issue could of caused this? if rebooted without an internet connection the date could revert to 1970 if using an unmodified SD card image.

What is your software setup on the Pi, Boelle ? Are you using the read-only SD image ? Do you have a permanent internet connection? Do you have any data input for that period ?

Paul

​

Somehow I lost a whole decade, that should be ~ 445w for minus 45years :-)

this has not to do with the kamstrup meter at all... it just the simple blinker on my electricity meter...

i'm using the timeseries engine as that allows me to do the intervals i want.

i'm still poking my mind how it can make missing transmits so much negative... 

"Although boelle hasn't shown his processing of that input, the input.png shows no additional processing icons, just "log" and "kwh""

thats right... i log the watt number as they come in because i want to graph my load over time

i then choose to process that number as kwh and log it also.... 

What is your software setup on the Pi, Boelle: Powered by openenergymonitor.org | vmaster:8.2.8, bufferedwrite:0.0.1

Are you using the read-only SD image ? no, trystan told me the best thing was to use the lowwrite image and if i wanted put data part of that image to a harddrive... since i had a 1TB drive arround i use that as data and the sd only yo boot on, but content wise the harddrive is the same as the sd image

 Do you have a permanent internet connection? yes 24 mbit down and 12 mbit up

Do you have any data input for that period ? not from that emontx no... but i did get from the other emontx that monitors my water meter, both are power from the same source, one connected to mini usb and then via 2 hookup wires to the other

pic of my processing attached

just wondering here... and i cant remember...

but if i did loose connection (my ISP tends to upgrade and reboot the cable router as they please) and i then reboot the pi to get wifi connection again it would cause it to revert to 1970's...

now i did put up a script that checks for connection and ping the router... the script runs every 5 mins, if no connection it will recreate it... that at least will reduce my need to reboot the pi if the router drops the connection.

but still it does not explain why it was only one emontx that did not get its transmits through in that timespan... i know that they are both powered... if one is the other is also, no bad soldering here... 

if it was only the reboot of the pi it should have been a drop in transmit from everything but in much shorter timespan as the pi only does very little loading from the sd and the rest from the harddrive...

now the negative value is slowly climbing towards 0 again so at least that tells me that its counting right again... and if i could just edit the data i could simply add the initial negative value to all datapoint and come up with somewhat ok data....

problem is that timeseries feeds does not show up in phpmyadmin... just the feednames... the old images a year back did allowed me to edit the data direct in phpmyadmin...

just thoughts here.... 

EDIT: small video of my setup in the utility closet (1 min long) http://youtu.be/o1Am2iLlCh8 (uploading right now)

what you dont see in the video is the 3 core cable that feeds through the fusebox then a hole out to the meter that is embedded in the wall on the outside just next to my door. also you dont see the 3 core cable that goes to the top of the hotwater cylinder. (brass pocket installed at factory)

Can you check your emonhub logfile for any 1970 dates using

cat /var/log/emonhub/emonhub.log

and scroll back looking at the dates down the left.

and also check if you are using a symlinked fake-hwclock file using this command

ls -la /etc/fake-hwclock.data

if it returns "/etc/fake-hwclock.data" ok but it may return something like "/etc/fake-hwclock.data -> /home/pi/data/fake-hwclock.data" and that means you may be vulnerable to the 1970 issue at boot up until a internet connection is established.

if it is this issue it would only need the clock to be out momentarily to register the time calculation in just the kwh log, the other feeds are less sensitive because it would just miss the "1970" time stamped data and look like it dropped one or more packets, but it only takes 1 "1970" timestamp to throw out the kwh log as the time since the last datapoint is used in every calculation.

Paul

yep--- loads of 1970 dates

1970-01-01 00:00:35,494 INFO EmonHub Pre-Release Development Version (rc1.0)
1970-01-01 00:00:35,503 INFO Opening hub...
1970-01-01 00:00:38,569 WARNING 1 Discarded RX frame 'non-numerical content' : [                                                                                                                                                             '>\x00', '8b']
1970-01-01 00:00:39,202 WARNING 4 Discard RX frame 'information' : ['>', '210g']
1970-01-01 00:00:39,632 WARNING 6 Discard RX frame 'information' : ['>', '15i']
1970-01-01 00:00:40,534 WARNING Send failure: wanted 'ok' but got Can't connect                                                                                                                                                              to redis database, it may be that redis-server is not installed or started see r                                                                                                                                                             eadme for redis installation
1970-01-01 00:00:41,154 WARNING Send failure: wanted 'ok' but got Can't connect                                                                                                                                                              to redis database, it may be that redis-server is not installed or started see r                                                                                                                                                             eadme for redis installation
1970-01-01 00:00:41,734 WARNING Send failure: wanted 'ok' but got Can't connect                                                                                                                                                              to redis database, it may be that redis-server is not installed or started see r                                                                                                                                                             eadme for redis installation
1970-01-01 00:00:42,315 WARNING Send failure: wanted 'ok' but got Can't connect                                                                                                                                                              to redis database, it may be that redis-server is not installed or started see r                                                                                                                                                             eadme for redis installation

the lines above was the complete screen output

lrwxrwxrwx 1 root root 31 Aug 13 08:50 /etc/fake-hwclock.data -> /home/pi/data/fake-hwclock.data

so i guess it was an 1970 issue after all...

2 Q's

1. is there anything i can do to prevent this happen to me again?

2. given that i cant access the feed data through phpmyadmin are there other tools i can use to edit the data so i can adjust the negative numbers to positive?

Run the following command in "read & write mode" (rpi-rw) to sort the time issue

    git clone https://github.com/emonhub/ntp-backup.git ~/ntp-backup && ~/ntp-backup/install

You don't want to "adjust the negative numbers to positive" as that would be just as inaccurate. you need to remove the 1 minus datapoint or add on an amount equal to the minus amount to balance it.

It's along shot, but if you can pin point the actual data point that caused the issue sat pm by looking at 

http://energymonitor-dk.dns4e.net/emoncms/boelle/vis/auto?feedid=78 or clicking on the kwh icon on the input page and zooming in to get the time and date which you can convert to UNIXTIME (UTC) and try using this url

http://energymonitor-dk.dns4e.net/emoncms/boelle/feed/deletedatapoint.json?id=78&feedtime=UNIXTIME

​to remove that one datapoint, however I don't know this will work for sure so be cautious.

(you will need to be logged in for the urls to work or add " &apikey=XXXXETC ")

Paul

remember feed 78 is made up on the basis of feed 77.... so all the datapoint from the time it did go negative and forwards are negative... hence my idea of adjusting them... and by adjusting i simple mean add 175477.5 to all datapoints from the drop up to now... 

as the datapoints  are working towards 0 again would make them positive by the amount i have consumed 

but again there are no way to edit datapoints... only delete them...

all datapoints in my watt log are all positive.... no need to mess with those

a pic on how the feed is going towards 0

If you've installed ntp-backup that should stop any future occurrences.

Going back and correcting historical data isn't easy as there's no current provision for that. It's not impossible but you need to work out how much effort it's worth to retain that data, it wasn't that long ago you reset the feeds.

You can either "just start over" - easy, less desirable option or wait until you find a way to "correct" all the effected data which is still being logged incorrectly. A third option could be to correct the current value on that feed now, so that from here on it is correct and if at a later date you find a way to correct the couple of days effected great, if not it's not so bad.

You will need to stop emonhub while applying the correction but it should only take a few seconds to complete. you will need to insert a new datapoint with the current unixtime and the new feed value (current feed value + correction)

1 - Stop emonhub using

sudo service emonhub stop

2 - Calculate the NEWVALUE by adding your correction value (175477.5 ?) to the value returned by

http://energymonitor-dk.dns4e.net/emoncms/boelle/feed/value.json?id=78

3 - get the current UNIXTIME from http://www.epochconverter.com/

4 - enter UNIXTIME and NEWVALUE to this url (all one line) to correct the current feed value

http://energymonitor-dk.dns4e.net/emoncms/boelle/feed/insert.json?id=78&time=UNIXTIME&value=NEWVALUE

5 - start emonhub with

sudo service emonhub start

well i might as well reset the kwh feed...

just a shame that timeseres can not be edited from phpmyadmin :-/

watt and kwh feed cleared of datapoints...

but for the next update i suggest a fix of some sorts that prevents this from happening again.. i might not benefit from that as i have taken measure against the need to reboot at all if just the wifi drops.

Hi boelle,

Yes. you will need to delete the feed or "just correct from now" and ignore the incorrect data. The penny has just dropped, as you are using the "buffered write" emoncms you cannot edit or delete past data, that was all disabled to achieve the low write levels required. you can only append data to the feed due to buffering.

The time issue is unfortunate, ntp-backup should fix it and hopefully it will get used in future read-only images too.