RESOLVED - Emon Base - problems uploading to the server

Ladies & Gents,

                             I have now assembled my EmonBase, EmonGLCD and EmonTx & all appear to be passing data between each other happily. Unfortunately, I can't get any data up to the web. The ethernet lights are on, the EmonBase picks up a DHCP address & resolves & displays the server IP address. Unfortunately, no data arrives at the web server.

Thinking my slightly tempermental O2 wireless router may be to blame, I followed the excellent instructions & set up an XAMPP emoncms server inside my home network - alas with the same result. To either server, clicking on the API links with a web broser sends test data quite happily. Unfortunately, the EmonBase data does not seem to arrive at either and does not receive a response from the server. Below (forgive the long windedness) is the code currently running - I have messed around with it a lot, but now gone back to standard for a sanity check.

Owing to a build error (my fault, I put the radio module on upside down the first time) I have a 'spare' EmonBase, so I have tried swapping the ethernet ICs. No effect, the problem is exactly the same.

The system is set up as type2 and the metered values are credible.

Running this code, please find below a sample from the serial port monitoring.

Any ideas what is wrong?

Thanks in advance

Matt

P.S. GLCD works fine, with both units receiving data from the EmonTx. On the one or two fleeting occasions the EmonBase has posted something to the web & received a response, the time on the EmonGLCD has updated properly - however that has only ever worked once or twice, usually straight after a reset.

 

/*                          _                                                      _     
                           | |                                                    | |    
  ___ _ __ ___   ___  _ __ | |__   __ _ ___  ___       _ __   __ _ _ __   ___   __| | ___
 / _ \ '_ ` _ \ / _ \| '_ \| '_ \ / _` / __|/ _ \     | '_ \ / _` | '_ \ / _ \ / _` |/ _ \
|  __/ | | | | | (_) | | | | |_) | (_| \__ \  __/  _  | | | | (_| | | | | (_) | (_| |  __/
 \___|_| |_| |_|\___/|_| |_|_.__/ \__,_|___/\___| (_) |_| |_|\__,_|_| |_|\___/ \__,_|\___|
                                                                                         
*/
//--------------------------------------------------------------------------------------
// Relay's data recieved by emontx up to emoncms
// Relay's data recieved by emonglcd up to emoncms
// Decodes reply from server to set software real time clock
// Relay's time data to emonglcd - and any other listening nodes.
// Looks for 'ok' reply from http request to verify data reached emoncms

// emonBase Documentation: http://openenergymonitor.org/emon/emonbase

// Authors: Trystan Lea and Glyn Hudson
// Part of the: openenergymonitor.org project
// Licenced under GNU GPL V3
//http://openenergymonitor.org/emon/license

// EtherCard Library by Jean-Claude Wippler and Andrew Lindsay
// JeeLib Library by Jean-Claude Wippler
//
// THIS SKETCH REQUIRES:
//
// Libraries in the standard arduino libraries folder:
//    - JeeLib        https://github.com/jcw/jeelib
//    - EtherCard        https://github.com/jcw/ethercard/
//
// Other files in project directory (should appear in the arduino tabs above)
//    - decode_reply.ino
//    - dhcp_dns.ino
//--------------------------------------------------------------------------------------

#define DEBUG     //comment out to disable serial printing to increase long term stability
#define UNO       //anti crash wachdog reset only works with Uno (optiboot) bootloader, comment out the line if using delianuova

#include <JeeLib.h>         //https://github.com/jcw/jeelib
#include <avr/wdt.h>

#define MYNODE 15           
#define freq RF12_868MHZ     // frequency
#define group 210            // network group

//---------------------------------------------------
// Data structures for transfering data between units
//---------------------------------------------------

typedef struct { int power1, power2, power3, voltage; } PayloadTX;
PayloadTX emontx;

typedef struct { int temperature; } PayloadGLCD;
PayloadGLCD emonglcd;

typedef struct { int hour, mins;} PayloadBase;
PayloadBase emonbase;
//---------------------------------------------------

//---------------------------------------------------------------------
// The PacketBuffer class is used to generate the json string that is send via ethernet - JeeLabs
//---------------------------------------------------------------------
class PacketBuffer : public Print {
public:
    PacketBuffer () : fill (0) {}
    const char* buffer() { return buf; }
    byte length() { return fill; }
    void reset()
    {
      memset(buf,NULL,sizeof(buf));
      fill = 0;
    }
    virtual size_t write (uint8_t ch)
        { if (fill < sizeof buf) buf[fill++] = ch; }
    byte fill;
    char buf[150];
    private:
};
PacketBuffer str;

//--------------------------------------------------------------------------
// Ethernet
//--------------------------------------------------------------------------
#include <EtherCard.h>        //https://github.com/jcw/ethercard

// ethernet interface mac address, must be unique on the LAN
static byte mymac[] = { 0x42,0x31,0x42,0x21,0x30,0x31 };

//IP address of remote sever, only needed when posting to a server that has not got a dns domain name (staticIP e.g local server)
byte Ethernet::buffer[700];
static uint32_t timer;

//Domain name of remote webserver - leave blank if posting to IP address
char website[] PROGMEM = "vis.openenergymonitor.org";
//static byte hisip[] = { 213,138,101,177 };    // un-comment for posting to static IP server (no domain name)           

const int redLED = 6;                     // NanodeRF RED indicator LED
const int greenLED = 5;                   // NanodeRF GREEN indicator LED

int ethernet_error = 0;                   // Etherent (controller/DHCP) error flag
int rf_error = 0;                         // RF error flag - high when no data received
int ethernet_requests = 0;                // count ethernet requests without reply                

int dhcp_status = 0;
int dns_status = 0;
int emonglcd_rx = 0;                      // Used to indicate that emonglcd data is available
int data_ready=0;                         // Used to signal that emontx data is ready to be sent
unsigned long last_rf;                    // Used to check for regular emontx data - otherwise error

char line_buf[50];                        // Used to store line of http reply header

//**********************************************************************************************************************
// SETUP
//**********************************************************************************************************************
void setup () {
 
  //Nanode RF LED indictor  setup - green flashing means good - red on for a long time means bad!
  //High means off since NanodeRF tri-state buffer inverts signal
  pinMode(redLED, OUTPUT); digitalWrite(redLED,LOW);           
  pinMode(greenLED, OUTPUT); digitalWrite(greenLED,LOW);      
  delay(100); digitalWrite(redLED,HIGH);                          // turn off redLED
 
  Serial.begin(9600);
  Serial.println("\n[webClient]");

  if (ether.begin(sizeof Ethernet::buffer, mymac) == 0) {
    Serial.println( "Failed to access Ethernet controller");
    ethernet_error = 1; 
  }

  dhcp_status = 0;
  dns_status = 0;
  ethernet_requests = 0;
  ethernet_error=0;
  rf_error=0;
 
  rf12_initialize(MYNODE, freq,group);
  last_rf = millis()-40000;                                       // setting lastRF back 40s is useful as it forces the ethernet code to run straight away
  
  digitalWrite(greenLED,HIGH);                                    // Green LED off - indicate that setup has finished
 
  #ifdef UNO
  wdt_enable(WDTO_8S);
  #endif;
}

//**********************************************************************************************************************
// LOOP
//**********************************************************************************************************************
void loop () {
 
  #ifdef UNO
  wdt_reset();
  #endif

  dhcp_dns();   // handle dhcp and dns setup - see dhcp_dns tab
 
  // Display error states on status LED
  if (ethernet_error==1 || rf_error==1 || ethernet_requests > 0) digitalWrite(redLED,LOW);
    else digitalWrite(redLED,HIGH);

  //-----------------------------------------------------------------------------------------------------------------
  // 1) On RF recieve
  //-----------------------------------------------------------------------------------------------------------------
  if (rf12_recvDone()){     
      if (rf12_crc == 0 && (rf12_hdr & RF12_HDR_CTL) == 0)
      {
        int node_id = (rf12_hdr & 0x1F);
       
        if (node_id == 10)                                               // EMONTX
        {
          emontx = *(PayloadTX*) rf12_data;                              // get emontx data
          Serial.println();                                              // print emontx data to serial
          Serial.print("emonTx data rx");
          last_rf = millis();                                            // reset lastRF timer
         
          delay(50);                                                     // make sure serial printing finished
                              
          // JSON creation: JSON sent are of the format: {key1:value1,key2:value2} and so on
         
          str.reset();                                                   // Reset json string     
          str.print("{rf_fail:0");                                       // RF recieved so no failure
          str.print(",power1:");        str.print(emontx.power1);          // Add power reading
          str.print(",voltage:");      str.print(emontx.voltage);        // Add emontx battery voltage reading
   
          data_ready = 1;                                                // data is ready
          rf_error = 0;
        }
       
        if (node_id == 20)                                               // EMONGLCD
        {
          emonglcd = *(PayloadGLCD*) rf12_data;                          // get emonglcd data
          Serial.print("emonGLCD temp recv: ");                          // print output
          Serial.println(emonglcd.temperature); 
          emonglcd_rx = 1;       
        }
      }
    }

  //-----------------------------------------------------------------------------------------------------------------
  // 2) If no data is recieved from rf12 module the server is updated every 30s with RFfail = 1 indicator for debugging
  //-----------------------------------------------------------------------------------------------------------------
  if ((millis()-last_rf)>30000)
  {
    last_rf = millis();                                                 // reset lastRF timer
    str.reset();                                                        // reset json string
    str.print("{rf_fail:1");                                            // No RF received in 30 seconds so send failure
    data_ready = 1;                                                     // Ok, data is ready
    rf_error=1;
  }

  //-----------------------------------------------------------------------------------------------------------------
  // 3) Send data via ethernet
  //-----------------------------------------------------------------------------------------------------------------
  ether.packetLoop(ether.packetReceive());
 
  if (data_ready) {
   
    // include temperature data from emonglcd if it has been recieved
    if (emonglcd_rx) {
      str.print(",temperature:"); 
      str.print(emonglcd.temperature/100.0);
      emonglcd_rx = 0;
    }
   
    str.print("}\0");  //  End of json string
   
    Serial.print("2 "); Serial.println(str.buf); // print to serial json string

    // Example of posting to emoncms v3 demo account goto http://vis.openenergymonitor.org/emoncms3
    // and login with sandbox:sandbox
    // To point to your account just enter your WRITE APIKEY
    ethernet_requests ++;
    ether.browseUrl(PSTR("/emoncms3/api/post.json?apikey=0888d6e4b3dcd51b45a1941f1c8b40aa&json="),str.buf, website, my_callback);
    data_ready =0;
  }
 
  if (ethernet_requests > 10) delay(10000); // Reset the nanode if more than 10 request attempts have been tried without a reply

}
//**********************************************************************************************************************

//-----------------------------------------------------------------------------------
// Ethernet callback
// recieve reply and decode
//-----------------------------------------------------------------------------------
static void my_callback (byte status, word off, word len) {
 
  get_header_line(2,off);      // Get the date and time from the header
  Serial.print("ok recv from server | ");    // Print out the date and time
  Serial.println(line_buf);    // Print out the date and time
 
  // Decode date time string to get integers for hour, min, sec, day
  // We just search for the characters and hope they are in the right place
  char val[1];
  val[0] = line_buf[23]; val[1] = line_buf[24];
  int hour = atoi(val);
  val[0] = line_buf[26]; val[1] = line_buf[27];
  int mins = atoi(val);
  val[0] = line_buf[29]; val[1] = line_buf[30];
  int sec = atoi(val);
  val[0] = line_buf[11]; val[1] = line_buf[12];
  int day = atoi(val);
   
  if (hour>0 || mins>0 || sec>0) {  //don't send all zeros, happens when server failes to returns reponce to avoide GLCD getting mistakenly set to midnight
    emonbase.hour = hour;              //add current date and time to payload ready to be sent to emonGLCD
      emonbase.mins = mins;
  }
  //-----------------------------------------------------------------------------
 
  delay(100);
 
  // Send time data
  int i = 0; while (!rf12_canSend() && i<10) {rf12_recvDone(); i++;}    // if can send - exit if it gets stuck, as it seems too
  rf12_sendStart(0, &emonbase, sizeof emonbase);                        // send payload
  rf12_sendWait(0);
 
  Serial.println("time sent to emonGLCD");
 
  get_reply_data(off);
  if (strcmp(line_buf,"ok")) {ethernet_requests = 0; ethernet_error = 0;}  // check for ok reply from emoncms to verify data post request

 

And the serial monitoring....

[webClient]
DHCP status: 1
IP:  192.168.1.95
GW:  192.168.1.254
DNS: 8.8.8.8
DNS status: 1
SRV: 213.138.101.177
2 {rf_fail:1}
DHCP status: 1
IP:  192.168.1.95
GW:  192.168.1.254
DNS: 8.8.8.8

emonTx data rx2 {rf_fail:0,power1:-3012,voltage:23672}
DHCP status: 1
IP:  192.168.1.95
GW:  192.168.1.254
DNS: 8.8.8.8
DHCP status: 1
IP:  192.168.1.96
GW:  192.168.1.254
DNS: 8.8.8.8
2 {rf_fail:1}
DHCP status: 1
IP:  192.168.1.96
GW:  192.168.1.254
DNS: 8.8.8.8
DHCP status: 1
IP:  192.168.1.96
GW:  192.168.1.254
DNS: 8.8.8.8
2 {rf_fail:1}
DHCP status: 1
IP:  192.168.1.96
GW:  192.168.1.254
DNS: 8.8.8.8

emonTx data rx2 {rf_fail:0,power1:-2923,voltage:23689}
DHCP status: 1
IP:  192.168.1.96
GW:  192.168.1.254
DNS: 8.8.8.8
DHCP status: 1
IP:  192.168.1.96
GW:  192.168.1.254
DNS: 8.8.8.8
2 {rf_fail:1}
DHCP status: 1
IP:  192.168.1.97
GW:  192.168.1.254
DNS: 8.8.8.8

emonTx data rx2 {rf_fail:0,power1:-2196,voltage:23671}
DHCP status: 1
IP:  192.168.1.97
GW:  192.168.1.254
DNS: 8.8.8.8
DHCP status: 1
IP:  192.168.1.97
GW:  192.168.1.254
DNS: 8.8.8.8

emonTx data rx2 {rf_fail:0,power1:-1445,voltage:23638}
DHCP status: 1
IP:  192.168.1.97
GW:  192.168.1.254
DNS: 8.8.8.8
 

 

 

glyn.hudson's picture

Re: RESOLVED - Emon Base - problems uploading to the server

 Good work getting the system up and running, I see your nearly there, emonTx data is being received successfully. I think I have spotted the probelem.

You have the emonBase setup to post to vis.openenergymonitor.org (our demo server). If you fancy you could test out posting data to here (you will need to create an account and put in the API key) to test before posting to your own server.

In order to post to your own server you will need to change the line 

char website[] PROGMEM = "vis.openenergymonitor.org";

to be the domain name of you're server. If the server is on a local network and doesnt have a domain name you will need to comment out the above line, and un comment the line below, inserting the IP address of your emoncms server on your local network: 

//static byte hisip[] = { xxx,xxx,xxx,xxx };    // un-comment for posting to static IP server (no domain name)   

you will also need to un-comment lines 33-24 in dhcp_dns.ino

//ether.copyIp(ether.hisip, hisip); // un-comment for posting to static IP server (no domain name)
//dns_status = 1; // un-comment for posting to static IP server (no domain name)

Don't forget to put in the write only API key from emoncms.
 

Oracle617's picture

Re: RESOLVED - Emon Base - problems uploading to the server

Hi Glyn,

              Thank you for your post. I have indeed created an account on your demo server - and started there! Alas it is not working... I went back to 'your code' for this post, to check it wasn't me 'making it worse' with all the messing around I had done. Alas neither server works reliably. If you have admin rights - I am 'Oracle617' on your demo server, so you should be able to see the very occasional posts of data? This is the 'odd one which gets through,' generally immediately after a '10 failures' reset?

I swapped the API key for the version I was running on my own server & made the edits you described above - alas they didn't talk either.

Kind regards

Matt

mattwire's picture

Re: RESOLVED - Emon Base - problems uploading to the server

Hi Matt,

I'm also unable to post to my local server but have not had time to fully debug yet.  I eventually got it working when posting to www.emoncms.org but it will not work when posting to my local server.  I use DNS resolution for both and can see that this works correctly in both cases.

 

At the moment I am wondering whether there could be a bug in the ethercard library or something but need to do some more tests to find out.  I have done experiments talking to another web server on my lan and it does get the response (404 not found) which makes it even more mysterious as to why nothing is getting to the emoncms local server.  Keep me posted with your progress - and see if you can get it to talk to www.emoncms.org first.

 

Matthew

TrystanLea's picture

Re: RESOLVED - Emon Base - problems uploading to the server

Hmm, the serial output is really strange:

DHCP status: 1

IP:  192.168.1.95
GW:  192.168.1.254
DNS: 8.8.8.8
DNS status: 1
SRV: 213.138.101.177
2 {rf_fail:1}

As it has now got an DHCP IP address and it has found the server you should not see the lines repeated again

DHCP status: 1
IP:  192.168.1.95
GW:  192.168.1.254

I cant think why this would be the case, it looks like everything else is ok.

Javierete's picture

Re: RESOLVED - Emon Base - problems uploading to the server

 I'm having this behaviour too.

Just for your information, 3 or 4 weeks ago had working perfectly and posting to my private server, but now I've updated the emonbase sketch and the server side and isn't working.

I though that was my own server fault so I've opened an account on emoncms.org and pointed the sketch with the correct write API key. But still no luck.

Here's the output of serial monitor:

DNS status: 0
2 {rf_fail:1}
DNS status: 0

emonTx data rx2 {rf_fail:0,power1:71,voltage:3264}

[webClient]
DHCP status: 1
IP:  192.168.1.19
GW:  192.168.1.1
DNS: 8.8.8.8
DNS status: 0
2 {rf_fail:1}
DNS status: 0

emonTx data rx2 {rf_fail:0,power1:72,voltage:3264}
DNS status: 0
2 {rf_fail:1}

 

mattwire's picture

Re: RESOLVED - Emon Base - problems uploading to the server

Did you get anywhere with this?  I am still unable to post to the local server on my network (www.emoncms.org works fine).  In fact the only local server it can communicate with is the gateway (which is also dhcp and dns) - it then receives tcp RST packets with source address of my emoncms server but actually originating from the gateway.

Using the web interface on the local server works fine for posting data.

I been doing packet captures on the network and can't see what's wrong, only that it clearly is wrong.  All packets look identical with the obvious exception of source/destination IP addresses but the local ones are never reaching the destination :(

Anyone got any thoughts?

Matthew

TrystanLea's picture

Re: RESOLVED - Emon Base - problems uploading to the server

Sorry about this, I think it was unclear documentation of the things that needed changing on on my part. I have just reordered both nanoderf sketches bringing all the config options to one place, I have tested it here on both my home server and on emoncms.org and it works fine.

There are now the following config lines at the top of the sketch:

// ethernet interface mac address, must be unique on the LAN
static byte mymac[] = { 0x42,0x31,0x42,0x21,0x30,0x31 };

// 1) Set this to the domain name of your hosted emoncms - leave blank if posting to IP address 
char website[] PROGMEM = "emoncms.org";

// or if your posting to a static IP server:
static byte hisip[] = { 192,168,1,10 };

// change to true if you would like the sketch to use hisip
boolean use_hisip = false;  

// 2) If your emoncms install is in a subdirectory add details here i.e "/emoncms3"
char basedir[] = "";

// 3) Set to your account write apikey 
char apikey[] = "YOURAPIKEY";

To set to a local server, set the ip address, the basedir, apikey and set:

boolean use_hisip = true;

EnergyRnR's picture

Re: RESOLVED - Emon Base - problems uploading to the server

 

 
I'm seeing the same thing. just added a print statement to confirm ethernet_error is zero[that's the '0' in the serial output below.
 Red LED stay on, in the emonbase and it just cycles through as below.
 
Not getting the "ok received" from emoncms.org. I have this same code working in another site though....
Also, the node is incorrect; It should be node 25....
 
 
 
Data sent: http://emoncms.org/api/post.json?apikey=&node=18&csv=480,766,1030,227
0
Data sent: http://emoncms.org/api/post.json?apikey=&node=18&csv=483,760,1022,227
0
 
[webClient]
DHCP status: 1
IP:  192.168.1.7
GW:  192.168.1.254
DNS: 8.8.8.8
DNS status: 1
SRV: 213.138.101.177
Data sent: http://emoncms.org/api/post.json?apikey=&node=18&csv=488,736,1028,226
0
Time request sent
Data sent: http://emoncms.org/api/post.json?apikey=&node=18&csv=489,764,1048,226
0
Data sent: http://emoncms.org/api/post.json?apikey=&node=18&csv=490,755,1030,226
0
Data sent: http://emoncms.org/api/post.json?apikey=&node=18&csv=488,746,1041,226
0
Data sent: http://emoncms.org/api/post.json?apikey=&node=18&csv=486,758,1042,226
0
Data sent: http://emoncms.org/api/post.json?apikey=&node=18&csv=483,755,1037,226
0
Data sent: http://emoncms.org/api/post.json?apikey=&node=18&csv=484,734,1033,226
0
Data sent: http://emoncms.org/api/post.json?apikey=&node=18&csv=484,748,1033,226
0
Data sent: http://emoncms.org/api/post.json?apikey=&node=18&csv=485,754,1032,226
0
Data sent: http://emoncms.org/api/post.json?apikey=&node=18&csv=481,750,1021,226
0
Data sent: http://emoncms.org/api/post.json?apikey=&node=18&csv=483,755,1025,226
0
 
[webClient]
DHCP status: 1
IP:  192.168.1.7
GW:  192.168.1.254
DNS: 8.8.8.8
DNS status: 1
SRV: 213.138.101.177
Data sent: http://emoncms.org/api/post.json?apikey=&node=18&csv=493,743,1033,225
0
Time request sent
Data sent: http://emoncms.org/api/post.json?apikey=&node=18&csv=501,751,1035,226
0
Data sent: http://emoncms.org/api/post.json?apikey=&node=18&csv=500,746,1036,227
0
Data sent: http://emoncms.org/api/post.json?apikey=&node=18&csv=502,757,1034,225
0
 
 
 
Eamonn
EnergyRnR's picture

Re: RESOLVED - Emon Base - problems uploading to the server

I should mention that the only change I have made from the working version is that I've dhcp_dns to include if (ether.dhcpExpired()) dhcp_status = 0; 

Previously I was using (!ether.dhcpValid()) 

Eamonn

EnergyRnR's picture

Re: RESOLVED - Emon Base - problems uploading to the server

I have swopped laptops so I suspect the version of ethercard.h I was using in the other laptop was older...hence I needed to update dhcp_dns..... ok - I'll stop having a soliloquy on the forum :-)

EnergyRnR's picture

Re: RESOLVED - Emon Base - problems uploading to the server

Sorted: just downloaded latest ethercard lib . Eamonn (I'm never moving laptops again!)

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.