Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error

Submitted by Dan Woodie on Fri, 12/07/2013 - 02:25

I am having problems getting my NanodeRF to properly communicate with my personal Emoncms server. I have my EmonTX transmitting valid data to my NanodeRF, which is seen on the serial port of the Nanode. The Nanode is properly receiving an IP address when I try in both the DHCP and in the static IP cases. Using packet sniffing I have determined that the NanodeRF is sending out a valid URL that is reaching the server, but the server is responding with a 403 Forbidden error indicating that it does not have permission to access that part of the server.

I have verified that the server is setup properly by copying the URL stream from the serial monitor and pasting it in a browser and getting the 'OK' response from the server. Using a packet sniffer, the only difference I can see between packets from my browser and packets from my Nanode is in the Host: section of the http GET. In my browser, the HOST section of the packet lists the proper sending IP address, while the packet from the Nanode shows garbage after the Host: text. This is occuring whether I use DHCP to set the Nanode IP address or I set it manually. Any ideas on what to check?

Dan

Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error

Submitted by dBC on Fri, 12/07/2013 - 08:11.

In my browser, the HOST section of the packet lists the proper sending IP address

Sending IP address, or the IP address of the server? I think the HOST field is how apache determines which virtual host to send the request to.

I'm not particularly familiar with the Nanode, but from a quick browse, it looks like this field:

// 1) Set this to the domain name of your hosted emoncms - leave blank if posting to IP address
char website[] PROGMEM = "emoncms.org";

could be what's supposed to end up in the HOST field.

Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error

Submitted by Mattia Rossi on Fri, 12/07/2013 - 08:41.

Also, could you check in your server logs what page is giving the 403 errors ?

Your server logs should also log the complete request (since it is a GET request) with no need of the packet sniffer.

Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error

Submitted by Dan Woodie on Fri, 12/07/2013 - 15:19.

Thanks for your assistance. I am using the NanodeRF_multinode_static_IP sketch (although I have tried and failed with all of the other sketches as well). Here are what I believe to be the relevant parts of the code. The API key has been hidden to protect the data. My local Emoncms server is 192.168.1.200 and my NanodeRF is the 192.168.1.201 address.

// ethernet interface mac address, must be unique on the LAN
static byte mymac[] = { 0x42,0x31,0x42,0x21,0x30,0x31 };

// 1) Set this to the domain name of your hosted emoncms - leave blank if posting to IP address
char website[] PROGMEM = "";

// or if your posting to a static IP server, change to true if you would like the sketch to post to static IP (not sure if this is working..)
boolean use_hisip = true; 
static byte hisip[] = { 192,168,1,200}; //emoncms.org IP

// Set to true and enter IP to give the NanodeRF a static IP address - default is DHCP (NB: must be used with static server IP (hisip)
boolean use_staticIP = true; 
static byte myip[] = { 192,168,1,201 };        //NanodeRF static IP address - not needed if using DHCP
static byte gwip[] = { 192,168,1,1 };        // gateway ip address - not needed if using DHCP

// 2) If your emoncms install is in a subdirectory add details here i.e "/emoncms3"
char basedir[] = "/emoncms";

// 3) Set to your account write apikey
char apikey[] = "***my API Key***";

My apache logs are not showing errors, possibly since the log level is not set properly. The access logs show the get request being received, but not an error. Here are the results of the packet sniffing, using Wireshark to follow the TCP data stream.

A GET request packet stream from the NanodeRF and the server response looks like this:

GET /emoncms/input/post.json?&apikey=***my API Key***&node=10&csv=2,0,43,3332 HTTP/1.0
Host:
Accept: text/html

HTTP/1.1 403 Forbidden
Date: Fri, 12 Jul 2013 14:50:42 GMT
Server: Apache/2.2.23 (FreeBSD) PHP/5.4.13 mod_ssl/2.2.23 OpenSSL/0.9.8x
Content-Length: 225
Connection: close
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /emoncms/input/post.json on this server.</p>
</body></html>

If I copy and paste the URL from the serial monitor, or from the packet stream and instead of sending it from the URL, send it from a browser I see this successful packet stream.

GET /emoncms/input/post.json?&apikey=***my API Key***&node=10&csv=2,0,42,3332 HTTP/1.1
Host: 192.168.1.200
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:12.0) Gecko/20100101 Firefox/12.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive

HTTP/1.1 200 OK
Date: Fri, 12 Jul 2013 14:50:39 GMT
Server: Apache/2.2.23 (FreeBSD) PHP/5.4.13 mod_ssl/2.2.23 OpenSSL/0.9.8x
X-Powered-By: PHP/5.4.13
Content-Length: 2
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html

The two packets are different in a variety of ways, but my main concern is that the Host section following the GET request is blank on the failing packet. When trying this with the NanodeRF_multinode sketch (using DHCP to assign the NanodeRF an IP address), I was getting the same errors, but in that case the Host section had garbage in it instead of being blank. So, that leads me to believe that this part of the ethercard code is not working properly, but I cannot verify. I am not sure if this is something that everyone has, but my server configuration is different and therefore not tolerant of, or if it is a sketch error. To test that I will later today setup an account on the main Emoncms server across the pond and see if I can post to that server with this sketch or if I get the same error.

Apologies for the weird web site formatting. I can't seem to figure out to make it wrap the long lines. Thanks in advance for any assistance.

Dan

Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error

Submitted by Robert Wall on Fri, 12/07/2013 - 15:45.

For what it's worth, I've only had success with a NanodeRF with a static dotted IP address (with WampServer under Windows). All your settings in the Nanode look sensible to me.

Is your server a separate machine, or is it running on the same machine as the browser?

When you sent the data from the browser, did emonCMS respond with "ok" (LOWER CASE, no quotes!) or do you mean Apache responded OK?

Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error

Submitted by Dan Woodie on Fri, 12/07/2013 - 17:33.

Robert,

My Emoncms is running on a dedicated server, on my home network, under FreeBSD. I posted a tutorial on getting it running under FreeBSD awhile ago. I am using a laptop to do packet sniffing on the network, and looking at the stream between the NanodeRF and the dedicated server. When I copy and post the TCP stream from the packet sniffing and use the browser on the laptop, I am able to see the OK response from Emoncms, and the data shows up in the feeds. I can also do that manually from other computers on the network. While I waited for my EmonTX kit to arrive, I used a professional 3-phase power analyzer (real nice setup I borrowed from work) to log some data from my home/solar panels. I took the spreadsheet of logged data in Excel and wrote a macro to post it to the server every 5 seconds to simulate realtime data being posted. During that work I played with the dashboard and visualization features in Emoncms, so I feel confident that the Emoncms is working properly.

My Dad is in town and we are hooking up the EmonTX to the circuits today and doing the calibration using the power analyzer. So, I will try and use the hosted Emoncms in an hour or two once our calibrations are done. Thanks for everyone's assistance.

Dan

Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error

Submitted by Dan Woodie on Fri, 12/07/2013 - 18:22.

Well, your server in Wales is receiving my data without error, so the problem is somewhere in my server configuration. I will start investigating that and determine what fixes it and post that here. If anyone has any ideas I would appreciate some hints since browser posted data is OK, but NanodeRF posted data is rejected. I will look at increasing the Apache logging and see if that points to anything.

Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error

Submitted by Dan Woodie on Fri, 12/07/2013 - 18:26.

On second look, packet sniffing the outgoing packets to Wales shows that the Host field of the packet is properly filled with "emoncms.org" so it may still be a NanodeRF issue. Anyone get the NanodeRF to work using a static IP address for the server?

Dan

Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error

Submitted by dBC on Fri, 12/07/2013 - 19:40.

Does it help any if you load this guy:

char website[] PROGMEM = "192.168.1.200";

with the static IP address of your server as shown?

That might get your Nanode request looking more like your browser request in the HOST field.

Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error

Submitted by Robert Wall on Fri, 12/07/2013 - 21:28.

As I said (maybe not too clearly), I have a NanodeRF working to Wampserver under Windows XP using static addressing. Here is the pertinent bit:

// ethernet interface mac address, must be unique on the LAN
static byte mymac[] = { 0x42,0x31,0x42,0x21,0x30,0x31 };

// 1) Set this to the domain name of your hosted emoncms - leave blank if posting to IP address
char website[] PROGMEM = "";

// or if your posting to a static IP server, change to true if you would like the sketch to post to static IP (not sure if this is working..)
boolean use_hisip = true;
//static byte hisip[] = { 213,138,101,177}; //emoncms.org IP
static byte hisip[] = { 192,168,1,65}; // IP of Machine with Wampserver running

// Set to true and enter IP to give the NanodeRF a static IP address - default is DHCP (NB: must be used with static server IP (hisip)
boolean use_staticIP = true;
static byte myip[] = { 192,168,1,66 }; //NanodeRF static IP address - not needed if using DHCP
static byte gwip[] = { 192,168,1,254 }; // gateway ip address - not needed if using DHCP

// 2) If your emoncms install is in a subdirectory add details here i.e "/emoncms3"
char basedir[] = "/emoncms5";

// 3) Set to your account write apikey
// char apikey[] = "----your API key ---";
char apikey[] = "b821f5936a56e44392c6f3bccf2bdcae"; // don't worry - this is a random string

//IP address of remote sever, only needed when posting to a server that has not got a dns domain name (staticIP e.g local server)
byte Ethernet::buffer[700];

That looks similar enough to me to think the problem isn't there - unless there's something funny going on with line endings or something like that.

Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error

Submitted by dBC on Fri, 12/07/2013 - 21:54.

My theory is (but it's nothing more than a hunch) that Robert's Wampserver is more forgiving about having a blank HOST field than Dan's apache is. From what I can tell (without have a setup here to experiment with), whatever you put in:

char website[] PROGMEM = "";

ends up in the HOST field. Since we know Dan's apache is happy when it sees 192.168.1.200 in the HOST field (as shown by his browser experiments), then perhaps forcing the Nanode to do the same may sort it.

Robert, do you have any way to inspect your packets to see if your HOST field is indeed blank with the above config?

Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error

Submitted by Robert Wall on Fri, 12/07/2013 - 23:09.

It will be late tomorrow (15 hrs at best) before I can look at that - assuming my brain doesn't overheat. What I can add now is I've looked at the library used by the Nanode - and it's totally uncommented and impenetrable. I think - I'm not sure - that if it sees anything in PROGMEM, it uses that in preference to the IP address, even if you say don't.

I'm still running Apache of course, it's just that the configuration is likely to be not quite the same! I've got Wireshark so I'll get that out.

Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error

Submitted by dBC on Sat, 13/07/2013 - 00:05.

Yeh, I think it very much depends on your virtual hosts config. There's a description of how apache uses the provided HOST: field here:

https://httpd.apache.org/docs/2.0/vhosts/details.html

... including which rule it follows when it's blank. So Dan, I suspect you've got two options:

1. find a way to force the Nanode to put the server IP address in the HOST field (like your browser does)

2. reconfigure your apache vhosts stuff so that blank or unknown HOSTs end up at the right virtual host

Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error

Submitted by Dan Woodie on Sat, 13/07/2013 - 02:06.

Thank you all for the comments and information. I have tried in the past to put the IP address in the website field as dBC indicated above, but that did not work at the time. When I tried that I did not have packet sniffing going on, just seeing no response from the server and no data posted. My guess is that the code would try to do a DNS lookup on that IP and get a garbage result back.

Thanks for your extra words Robert. I have tried all of those items as well with no success. Static IP for NanodeRF with static IP for server, DHCP for NanodeRF (successful) with static IP for server, and all variations of the same. I have also tried using my dynamic DNS address that points to my WAN IP address, and turning on port forwarding to my server, so the packet leaves my house, bounces back, and then hits the server using name addressing. The DNS lookup correctly gets my external IP address, but the server still refuses it.

One challenge is that without packet sniffing, or digging into the Apache logs, you cannot tell the difference between the NanodeRF not reaching your server, and it getting refused for some other reason. The serial monitor says the same thing. It may be useful if the NanodeRF gets some form of response back to post that response code instead of only posting the Data Received message on a successful post.

Looking through the documentation on the Apache httpd.conf file, there is a spot where you list the server name, or the server IP address. I have edited that many ways as well without success. I am also attempting to get my router, which can act as a local DNS server, to give the Emoncms server a network name that I can try to address, but so far have not been successful at that. Anyhow, I will dig into the Vhosts link you have given later this weekend. I will be busy most of tomorrow so I won't get back to this until tomorrow night or Sunday. Thanks again for your collective thinking and hopefully when we figure it out we can get a root cause to share and help ease the collective burden. Good night.

Dan

Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error

Submitted by Mattia Rossi on Sat, 13/07/2013 - 12:02.

Hi, had a look at the ethercard code that the firmware is using .... If you have set up apache with a virtual host identified by a name, then you should put that name in the website array, and you should halso put the ip in the hisip array and set the boolean use_hisip to true

If on the other hand, your virtualhost is identified by an ip, you should put the ip, and still use the hisip in order not to trigger a dns resolution

If it is neither of these two, then you should post the relevant part of the apache config file (the virtualhost section) to see whether you are doing something weird with the folder access rules....

Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error

Submitted by Robert Wall on Sat, 13/07/2013 - 14:54.

Here is the NanodeRF successfully talking to Apache under Wampserver. I've given you the bytes for the whole frame so that you can see exactly what's being sent and returned.

0000  00 14 a4 23 a4 3d 42 31  42 21 30 31 08 00 45 00   ...#.=B1 B!01..E.
0010  00 af 9f bf 40 00 40 06  16 b7 c0 a8 01 42 c0 a8   ....@.@. .....B..
0020  01 40 0b 53 00 50 00 00  9d 01 40 88 73 a0 50 18   .@.S.P.. ..@.s.P.
0030  04 00 c0 14 00 00 47 45  54 20 2f 65 6d 6f 6e 63   ......GE T /emonc
0040  6d 73 35 2f 69 6e 70 75  74 2f 70 6f 73 74 2e 6a   ms5/inpu t/post.j
0050  73 6f 6e 3f 61 70 69 6b  65 79 3d 35 30 30 35 65   son?apik ey=5005e
0060  30 66 39 32 63 35 37 64  34 35 63 35 64 33 63 38   0f92c57d 45c5d3c8
0070  35 63 32 62 31 38 34 32  66 61 35 26 6e 6f 64 65   5c2b1842 fa5&node
0080  3d 31 30 26 63 73 76 3d  30 2c 37 2c 32 35 31 35   =10&csv= 0,7,2515
0090  35 2c 30 2c 30 20 48 54  54 50 2f 31 2e 30 0d 0a   5,0,0 HT TP/1.0..
00a0  48 6f 73 74 3a 20 0d 0a  41 63 63 65 70 74 3a 20   Host: .. Accept:
00b0  74 65 78 74 2f 68 74 6d  6c 0d 0a 0d 0a            text/htm l....

0000  42 31 42 21 30 31 00 14  a4 23 a4 3d 08 00 45 00   B1B!01.. .#.=..E.
0010  00 e5 a0 06 40 00 80 06  d6 39 c0 a8 01 40 c0 a8   ....@... .9...@..
0020  01 42 00 50 0b 53 40 88  73 a0 00 00 9d 88 50 18   .B.P.S@. s.....P.
0030  3f ed 46 a6 00 00 48 54  54 50 2f 31 2e 31 20 32   ?.F...HT TP/1.1 2
0040  30 30 20 4f 4b 0d 0a 44  61 74 65 3a 20 53 61 74   00 OK..D ate: Sat
0050  2c 20 31 33 20 4a 75 6c  20 32 30 31 33 20 31 34   , 13 Jul  2013 14
0060  3a 31 33 3a 34 30 20 47  4d 54 0d 0a 53 65 72 76   :13:40 G MT..Serv
0070  65 72 3a 20 41 70 61 63  68 65 2f 32 2e 32 2e 32   er: Apac he/2.2.2
0080  31 20 28 57 69 6e 33 32  29 20 50 48 50 2f 35 2e   1 (Win32 ) PHP/5.
0090  33 2e 31 30 0d 0a 58 2d  50 6f 77 65 72 65 64 2d   3.10..X- Powered-
00a0  42 79 3a 20 50 48 50 2f  35 2e 33 2e 31 30 0d 0a   By: PHP/ 5.3.10..
00b0  43 6f 6e 74 65 6e 74 2d  4c 65 6e 67 74 68 3a 20   Content- Length:
00c0  32 0d 0a 43 6f 6e 6e 65  63 74 69 6f 6e 3a 20 63   2..Conne ction: c
00d0  6c 6f 73 65 0d 0a 43 6f  6e 74 65 6e 74 2d 54 79   lose..Co ntent-Ty
00e0  70 65 3a 20 74 65 78 74  2f 68 74 6d 6c 0d 0a 0d   pe: text /html...
00f0  0a 6f 6b                                           .ok

I think this just confirms, Dan, that your Nanode is doing the same as mine. It looks to me as if Matt is on the right track...

(Falsified APIkey again!)

Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error

Submitted by Dan Woodie on Mon, 15/07/2013 - 01:34.

Yes, I believe it is a server name issue. I recall when working to get the system originally running I had two different configuration files where the server name was put in, but I can only find one right now. I will dig further in and report back when I figure it out. In the mean time, I am logging my data to the Wales hosted Emoncms server to do some power logging so I can see how accurate my calibrations on the EmonTX have been. I have limited use of a very nice Fluke 435 3 phase power analyzer from work so I am trying to compared summed values over time to see how accurate it runs compared to the Fluke. The generated readings are fluctuating quite a lot so I am interested in seeing if that noise integrates out. Once I see that accuracy over a few days I will try out the Emonlibpro that was recently posted to compare.

Thanks for your notes and information. I will update when I figure more out.

Dan

Re: Solved - Problem with Nanode RF reaching personal Emoncms server - 403 Forbidden error

Submitted by Dan Woodie on Wed, 17/07/2013 - 03:13.

I have determined the problem and corrected it. The issue relates to addressing Emoncms servers only by IP address instead of by DNS name. The httpd.conf file used by the Apache server to set many server options has a section in which you tell it the server name. More information on it can be found here: http://httpd.apache.org/docs/2.2/configuring.html. The comments in the config file tell you if your server does not have a name, to put the IP address in it. But, even with that setting, it will look for the HOST: field of the GET packet to list that IP address before serving any files. Without that field being filled in, and nothing extra in the httpd.conf file to tell it what to do, it will return a HTTP 403 Forbidden code which the NanodeRF does not know what to do with.

By adding a virtual hosts section to the httpd.conf file, a server can serve multiple sites from one IP address, matching the hosts field with the server name for each host (http://httpd.apache.org/docs/current/vhosts/examples.html). In this case, we can use the virtual hosts section with a wildcard "*" to tell it what to do with packets that have the HOST field blank. So, adding:

<VirtualHost *:80>
DocumentRoot "/usr/local/www/apache22/data" # default directory for Apache files on FreeBSD
</VirtualHost>

to the httpd.conf file right after the default document root section of the config file fixed it. The DocumentRoot is simply the default root for the apache server as a whole. The *:80 in the VirtualHost header tells it what to do if a packet arrives that does not match any other listed virtual hosts. Since I don't have any other hosts listed, it defaults to the root for all packets.

So, to summarize, if you want a static IP for your personal Emoncms server, and you are getting 403 forbidden errors (which the NanodeRF will not report but you may see in the Apache logs if you set it to list the details (http://www.cyberciti.biz/faq/apache-logs/), set the ServerName to your static IP and add the above lines below the DocumentRoot section, and it should work.

Let me know if you have any questions. Thanks to all who assisted and for my Dad who tracked down the page on virtual host settings.

Dan

Archived Forum