Archived Forum

 Hi Baz,

The Pachube example is very much in a alpha stage of development as stated in the github readme. 

It will only self reset if using the 'Uno' bootloader and if #define uno is active. If using the duemilanove bootloader you should comment out #define uno to disable the reset watchdog. 

There are some issues with phasing the http responce from pachube, ongoing thread here: http://openenergymonitor.org/emon/node/363

Please send us a git pull request if you get it working. Our developments are mainly focused with posting to our own open-source energy web-app emoncms. The emonBase example for posting to emoncms are more developed. 

 There have been many discussions lately regarding emonBase Nanode/NanodeRF instability.

To constructively try and move forward, lets create a knowledge base of setups that are stable Vs. setups which are not. Hopefully this will help us to debug this issue and give us a framework in which to discuss problems.

See forum thread: http://openenergymonitor.org/emon/emonbase/stability

I think line 108 should be changed to be:
get_header_line(1,off); // Get the date and time from the header
if (strcmp(line_buf,"HTTP/1.1 200 OK")) {Serial.println("ok recieved"); request_attempt = 0;}

Since the reply from Pachube is:

HTTP/1.1 200 OK
Date: Tue, 31 Jan 2012 13:04:36 GMT
Content-Type: text/plain; charset=utf-8
Connection: close
X-Pachube-Logging-Key: logging.xxxxxxxxxxxxxxxxxxxxxxxxx
X-PachubeRequestId: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Cache-Control: max-age=0
Content-Length: 1
Age: 0
Vary: Accept-Encoding

Please report experiances. 

 I tried this (as suggested by Trystan) here http://openenergymonitor.org/emon/node/363 but it had no effect whatsover, with nothing, errors or otherwise,  being reported via the serial monitor.

The serial monitor still continued to post it's normal HTTP/1.1 200 OK ect response, but nothing further.

It's almost as if the my_callback command is not being called correctly via the line;
website, PSTR(FEED), website, PSTR(APIKEY), stash.size(), sd, my_callback);

Paul

 Another type of error?

Using the existing sketch on Gihub, I can successfully post to Pachube, and have ammended it to post my import/export reading, PV generated, and a calculated consuming.

However, after a "while" the feed stops being accepted, Errror 400 and "CSV Parser Error: CSV is invalid. Incorrect number of fields.". No further posts are then accepted until a reboot of the Nanode.

When I say a "while", it is a matter of a few minutes (2-10). Ive tried varying the rate at which I post to Pachube, and a shorter post interval makes the Error 400 quicker, but posting at 2 per minute still fails eventually.

Any thoughts? Ive also noticed that the my_callback routine doesnt actually seem to get called, even with it added to the end of the Stash. Am I right? (Nanode5).

Ive copied the serial output below, where it goes bad, with a couple of good posts before it./



RF recieved 1050.00Generating:120.60,ImpExp:1050.00,Consuming:1170.60

REQUEST: 200

PUT http://api.pachube.com/v2/feeds/42644.csv HTTP/1.0

Host: api.pachube.com

X-PachubeApiKey: p26blah0ZOSnlvaz0g

Content-Length: 32

0,120.60

1,1050.00

2,1170.60

RF recieved 1054.18Generating:120.60,ImpExp:1054.18,Consuming:1174.78ÿÿÿÿÿÿ

RF recieved 1045.76Generating:120.60,ImpExp:1045.76,Consuming:1166.361"

RF recieved -113.23Generating:113.23,ImpExp:1045.76,Consuming:1158.99ÿÿÿÿÿÿ

REQUEST: 200

PUT http://api.pachube.com/v2/feeds/42644.csv HTTP/1.0

Host: api.pachube.com

X-PachubeApiKey: p26n0blahxd0ZOSnlvaz0g

Content-Length: 32

-PachubeRequestId: 9c2742324e805

RF recieved -113.48Generating:113.48,ImpExp:1045.76,Consuming:1159.241"

RF recieved -112.91Generating:112.91,ImpExp:1045.76,Consuming:1158.67

REQUEST: 200

PUT http://api.pachube.com/v2/feeds/42644.csv HTTP/1.0

Host: api.pachube.com

X-PachubeApiKey: p26n0BsblahYxd0ZOSnlvaz0g

Content-Length: 32

o1xEVp

X-PachubeRequestId: 3013

REPLY:

HTTP/1.1 400 Bad Request

Date: Sun, 12 Feb 2012 15:26:01 GMT

Content-Type: text/plain; charset=utf-8

Connection: close

X-Pachube-Logging-Key: logging.y2Vp42Z0ldzaxso1xEVp

X-PachubeRequestId: ddfd528beeab90bea6f891d55f77058d481b1074

Cache-Control: no-cache

Content-Length: 61

Age: 0

CSV Parser Error: CSV is invalid. Incorrect number of fields.

Hi Andy

I had pretty much the same problem which seems to be pretty much fixed by resetting the flags every now and then (or rebooting the nanode) - so far 5 days without a single crash here. It also seems UNO bootloader is a bir safer to run than the original nanode's duemilanove. 

Check out this post for further details:

http://openenergymonitor.org/emon/node/363#comment-2993

Igor

 ok, so Ive flashed the Nanode5 with the Opti bootloader, and uploaded the sketch again.

Still no joy - there seems to be a fundamental problem that the my_callback routine is never called. (despite adding it to the end of the line : website, PSTR(FEED), website, PSTR(APIKEY), stash.size(), sd, my_callback)

As a result, request_attempt never gets reset to zero, and therefore always does a reset when the count (currently 10) is exceeded.

Ive put a print statement in the callback, and it never prints, so i know its not being called.

Am I missing something basic?

(At least if I set the counter to a reasonable value, I know I will always reset, but thats not the right answer).

 Andy, I found exactly the same and asked the question about it being called in my post (above). 

I'm probaby showing my lack of programming skills here, but it seems to me that the my_callback 'call' is rolled into the stash, which although is posted OK over the ethernet, does not call and run the local routine.

I also replaced the functions within my_callback with a Serial.print, which as you have also noticed does not print!

 Yes, the code never seems to get run locally - I tried setting some other variables, and printing them both in and outside of the routine - they're never set.

So Im wondering how the guys in the main thread (fjhug, Igor) are able to use other methods, as request_attempt never gets set??

 I don't think that they are getting automatically reset. Igor comments above that he 'resets the flags every now and then', so they don't appear to be working as per the sketch.

 Did we ever get to the cause of the actual problem?

Is it the code in the Nanode (Ethernet most likely?) causing a corruption? Im sure its programatic, as, if I had the patience to count, I thinks it vaguely related to the number of posts.

If I 'spam' Pachube with a post every second, I get the "400 Error"s approximately twice as quick as if I post every 2 seconds. The longer the distance between posts, the longer it take to get the error etc. 

If I have time tomorrow, I might sit and count......

 From my experience, the problems arise when the Pachube api server develops a problem, and the Nanode doesnt seem to be able to reconnect without a reboot.

I have confirmed this by searching for frozen feeds via the Pachube website when my Nanode crashes, and on each occassion I have found loads of other feeds which have frozen at exactly the same time. 

I contacted Pachube about this, and they confirmed to me that they are having intermittent server problems, and are in the process of migrating their data over to a new dedicated server to increase reliability and reduce downtime.

 I'll setup a project on PacHube in the next few weeks, using NanodeRF.

The my_callback function is called when the request returns. I'll try to dig a bit to see why the request wouldn't return.

I'll log the server reply from PacHube to see if it can be part of the problem. I had no issue posting to emoncms (local and beta). It might be that the condition to reset the request_attempt is different on emoncms than on PacHube. I'll try to implement processing the http status code from get_header_line(1,off); in my_callback, with the actual reply data coming back from the request, and update error flags accordingly.

I'll keep the post updated as I go along...

 Fails at 120 ish posts.

Ok, so Ive been trying to do more debugging.

Just to recap the position :

1.I can post virtually error free to EmonCMS, both locally and hosted.

2. The Pachube sketch works fine for a bit, but then starts returning Error 400 (which is not documented by Pachube) and various incorrect field format errors in the sketch serial output.

3. I had wondered if I was trying to flood Pachube, but the allowable rate is 100 per minute.

4. The SD_Callback function doesnt seem to work, so I used a the coded reset after so many posts.

I tried to determine what the number of posts or time was, when the errors occured, in case there was a correlation.

It doesnt seem as though there is any dependancy on how often (frequency) of posting is, any reasonable time frame.

However, whether I post at 60 per minute or 6 per minute, the errors occur after about 120 posts (range observed 110-140). Ive determined this simply by incrementing request_attempts, display it, and observing both the serial output and watching for an Error code of 400 in the Pachube Api Debug screen. I have inserted varioys DELAY values of between 500 and 8800 ms, and the results are consistent.

[Is there any way to code a higher DELAY value without the reset function kicking in?]

So I now know that I can get about 15 minutes (using delay of 8800) without a reset. Unfortunately, I want something nearer realtime (e.g 1S) which means a much shorter reset period - about 1 and half minutes. Also, unfortunately, this means I lose data for the period of the reset.

I have tried a couple of Nanodes, so its not a build problem, but it does lead me to conclude that perhaps the ethernet buffer is getting corrupted. 

Im no nearer to a programatic solution, so please continue to investigate, while I mess about!

 I'm not sure that it's a fair test if you are hammering data at Pachube at such a high rate, as other factors will then influence its stability, such as delays/bottlenecks in the network transmission/servers, and you may be transmitting the next batch of data before you have received a reply from the last.

To introduce a delay without influencing the watchdog reset, you could try a time related 'IF' statement, something like this for creating a 5 second delay and include the code that you wish to delay in the statement;

if ((millis() - prevMillis) > 5000) {
prevMillis=millis();
//The code or function that you want to delay goes here
}

This way, the loop keeps running and resetting the watchdog, and jumps over the 'IF' condition unless more than 5 seconds has elapsed.

 Playing with pachube, and because the sketch uses a different method to send data than emoncms (ether.tcpSend instead of ether.browseURL), the call back function is never called. This means the request_attempt is never reset. So in theory, the board will reset after the number set in the if statement before resetting error codes or waiting for watchdog.

Trying to make a sketch that can send to several destination (I would need local emoncms, remote emoncms and pachube). Need to dig into the EtherCard library, but at the end of the Pachube send, I reset the request_attempt. Will monitor this weekend...

Pachube working with POST.

Posting to pachube and emoncms from same NanodeRF for a week without problem.

I wrote a different sketch to send my data to pachube, and use POST instead of PUT from the pachube example provided. This gives me the control of the callback function.

It also implements server ports, as my local emoncms is running on port 8888.

Ended making a sketch to send to pachube and/or local emoncms and/or emoncms beta from one Nanode.

Sketch attached.

Any feedback or test to pachube would be welcome... Thanks

Hi Francois

This sounds fantastic. We will give it test then put it on on the OEM github. Has anyone else given it a go? Please post up experiences.

Using POST instead of PUT to gain access to the callback is very interesting. We cam to the conclusion that the callback would not be accessible from Pachube without modifying the EtherCard library.  

Good work! 

Going to give this a go...

 ....hmm getting 401 Unauthorised error and cant immediately see the problem,

Have change my feed and API, and commented out the Local and remote Emon CMS calls.

The IP stuff is resolving ok, but actual post is failing.

 This is usually a wrong write api key / feed combination. I had this error when not setting up the correct api key.

Make sure your api key has full access to the feed, and that it is not restricted in the methods.

You can check the requests coming in pachube in the Debug info of your account in the help section. You will see the request and the reply sent by pachube. 

 Yes - schoolboy error/success

Cut and Paste had left a random character in my key.

The good news is this looks like its working for me.

Ive already gone straight past my 120 post limit, and will let it run overnight.

Pachube feed 42644/Posts has a record of the number of successful posts. Please ignore anything prior to 21:00 GMT.

[manual reset for code update at 13:00 24/02 GMT after nearly 8000 successful posts]

Im also logging to VIS.EmonCMS concurrently.

I'll tempt fate and say "good work" Francois - thanks.

I was getting similar behaviour due to the CSV data or Pachube API becoming corrupted. This seemed to be worse when using Pragma / Flash strings. To help see the problem you can connect your nanode to your PC via Internet Connection Sharing and look at the actual TCP data being sent by using WireShark.

Hi François

I'm using your previous suggestion (resetting the error flags every few hundred cycles) and my Nanode hasn't had a single crash for over 3 weeks. I think I'll try the new sketch with my other nanode and compare the results.

And thanks for your contribution! It saved me a lot of time! :)

Igor

 I'll post this here as it may be useful:

I've been posting to pachube for a while now with a nanode-RF. I use 

EtherCard. 

The line that sends it is: 

        ether.httpPost( PSTR(PACHUBEAPIURL), pachubeFeedPath, PSTR(PACHUBE_VHOST),          PSTR(PACHUBEAPIKEY), pachubeData ,&browserresult_callback); 

I am using the v2 api with urls similar to : /v2/feeds/12345.csv?_method=PUT 

the /v2/feeds/ part is defined by PACHUBEAPIURL as its fixed, the rest is set in pachubeFeedPath.  PACHBE_VHOST and the key are another pair of #defines. 

the pachubeData is just a string built up of the data e.g.: 

0,1234 
1,10.4 
2,123 

etc. 

This means I can use one request to post updates for multiple streams on the same feed. 
The callback function is executed when the response is received, and all I do in here at present is turn the green LED off as it was turned on before calling the httpPost function. It serves as an indication that something has happened. 

Hope this helps 

SRB

Glad to help Igor. Just started in the opensource world. I guess this is what collaborative projects are about.

Love the project. Great work

Still posting without issue to pachube and emoncms. Just need to start monitoring energy now... this is why I got into it after all!

Pachube feeds available at https://pachube.com/feeds/49831. Some interruptions are caused by sketch updates. Have another NanodeRF posting to private feed to test stability.

When I try to compile (Arudnio 022) NanodeRFmulit I get this error

NanodeRFmulti:77: error: conflicting return type specified for 'virtual size_t PacketBuffer::write(uint8_t)'

It's having a problem with the bolded line below

class PacketBuffer : public Print {
public:
    PacketBuffer () : fill (0) {}
    const char* buffer() { return buf; }
    byte length() { return fill; }
    void reset()
    {
      memset(buf,NULL,sizeof(buf));
      fill = 0;
    }
    virtual size_t write (uint8_t ch)         
      { if (fill < sizeof buf) buf[fill++] = ch; }
    byte fill;
    char buf[150];       
    private:
};
PacketBuffer str;

I figured it out by looking at the code in packetBuf.ino.  Code is differnet for Arduino 1 vs 022.

I added:

#if ARDUINO < 100
    virtual void write(uint8_t ch)
    {
    if (fill < sizeof buf)
      buf[fill++] = ch;
    }
#else
    virtual size_t write (uint8_t ch)
    {
     if (fill < sizeof buf)
      buf[fill++] = ch;
    }
#endif

Now it compiles

I'm having some problems comparing the string returned in get_header_line in the callback_pac() function.  First, I think the strcmp() code is wrong because if the two strings are equal, strcmp return zero; and the code in the if statement won't be executed.

  get_header_line(1, off); 
  if (strcmp(line_buf,"HTTP/1.1 200 OK")){...
 

So I changed it to:

  get_header_line(1, off); 
  if (strcmp(line_buf,"HTTP/1.1 200 OK") == 0){...

But strcmp still doesn't think the strings are equal so it doesn't return zero.  When I print line_buf to the serial monitor, its: HTTP/1.1 200 OK.  So the seem to match.  Could there be an issue with null terminator or something that's causing this problem?

I figured out what's going on.  In the text returned by pachube, there is a return character before the end of line charachter.  So pachube is sending this:

HTTP/1.1 200 OK\r\n

and get_header_line() is returning:

HTTP/1.1 200 OK\r\0

I changed the strcmp to this (\r at the end):

if (strcmp(line_buf,"HTTP/1.1 200 OK\r") == 0)

Now iot works fine.

 Well done. I'll change my sketch at the week end to be able to process various codes returned. I'll keep the post updated.

Can someone explain what these two lines do.  I'd guess the first one just loops until ether.packetLoop doesn't equal 0.  Is the 2nd line reading one byte of data?

while (ether.packetLoop(ether.packetReceive()) != 0) {}

ether.packetLoop(ether.packetReceive());

From the library, the while statement waits for the ethernet buffer to be empty (on receive or send).

the packetLoop is supposed to make sure no background low level activity  (like ping request...) is on going before using the buffer.

Haven't looked into it much more than that, so it might be very high level explanation !

 This is really coming together now, and it's good to see the contribution that Francois has made in getting this to work.

I'm still resetting my board every every hour via a simple timer & AVR watchdog, which although crude, works! However when I get some time I want to update the code as per this thread.

One question though, rather than use Francois's full sketch (as I have so many other things in the sketch), would it be possible to amend just this part of my existing sketch, shown here;

to POST instead of PUT, so that it will call the my_callback function (which I can add myself)

 You can use your sketch, and provided the ip and dns bits are done, just use the following lines to send the str string:

 Francois, It may be a few days before I get time to make the changes, but thank you for your reply and help.

As you will see from my previous forum posts I have tried to get this to work for months without success, and it's great that you have brought your skills to the forum.

I wish I had seen this thread earlier, it might have saved me some time. Anyway here is a simple NanodeRF sketch for posting to COSM that is compatible with the current emontx payload format in GitHub. Just patch in your COSM feed id and api key into the call to httpPost and make sure your COSM DataStream names match the ones hard coded in the csv data.

Roger

Another version of my COSM sketch. This is very similar to fjhug's NanodeRFMulti sketch. I needs a small mod to tcpip.cpp in the JeeLabs EtherCard library. Just remove the static from tcp_client_state. This means I can can check for the tcp connection being properly closed before I go on to open a new one. A couple of observations about the Ethercard library;-

1. It does not handle out sequence tcp PDU fragments. I seem to get a lot of these from COSM. If the out sequence PDU happens to have the FIN flag set then the library will close the connection and discard data before I get a chance the see the HTTP 200 PDU. This causes timeouts and weirdness, but the sketch always seems to recover.

2. It cycles round a very small (255) number of source ports. This means that source ports in syns quickly come round again. This can confuse NAT gateways.

Enjoy,

Roger

hey roger,

nice post from your cosmemon scripts. i'm trying to make an nanodeRF work with and sensorshield and posting to emomcms and cosm. But I have difficulties with the call_back ..so I will try your script.

but it's give an error on the tcp_client_state. I probably have to mod. the tcpip.cpp for it (I read your comment) but what do I have to modify there?

sollution: alter the tcip.cpp file line 32 with command out [//static byte tcp_client_state;]

save it, restart arduino IDE and load your sketch.

Hi Guys,

Thanks a lot for making good progress on the Pachube/Cosm example. Our development efforts have been focused around emoncms but there is obviously value in having a working NanodeRF Cosum example. I have just tested the code posted by Roger James. After a little tweak to support a change in the EtherCard API it worked first time. I have pushed this example to github as our NanodeRF Cosum example: https://github.com/openenergymonitor/NanodeRF

I have tried to credit those involved, please let me know if I could do this better in any way. 

Thanks again,

Glyn.