27 close();
28 op=0;
29 dcfan("on");
30 delay(sample_time);
31 dcfan("off");
32 data="";
33 }
34 }
35 }
where we initiate the I2C wire ports as slave in address 9 and make it wait for data. Then we proceed to attach a function to the trigger associated with the data reception event.
In this function, we basically transcribe the serialized char packets sent and reconstruct the original string message. Meanwhile, we have the main slave loop running. In this snippet we show the most important part of our loop function that is the detection of the information to measure and according procedure. In our actual deployed code we have several more functions to operate individual actuators and show status of operation through LEDs, among other features, but those end up serving more of debug tools. In this loop, we wait for the message to measure, that is the char m as we have seen in the master code. When this happens, the slave knows it needs to make a measurement however it waits for information on sampling time. When this information arrives, the slave starts the procedure and invokes many of the functions already shown in other snippets of code. Nevertheless, the procedure is: open sample holder, activate stepper to unroll filters, close sample holder, turn on the DC fan, wait sampling time, turn offthe DC fan and go back to waiting for a measurement.
B.6 Server communication via GSM/GPRS
GSM communication and server setup was the hardest part of the development of this device. There were plenty of difficulties in the task of connecting this device to a server that we could control. The major one stems from the fact that it is being done by GSM/G-PRS instead of the much more straightforward Ethernet cable connection or even the Wifi wireless connection that would still be much easier to implement. The drawbacks of GSM/GPRS are numerous from the lack of support, to the difficulty to operate within a plant facility that acts like a Faraday cage due to its metallic construction and makes GSM cell reception unreliable at best. Nevertheless, given the fact that corporate secrecy might be compromised by giving this device assess to the plant internal network we had to pursue this solution.
The correct deployment of both a client and server took several iterations and eventu-ally spread across a few months. Not only it was needed to build a server from scratch, but also a client program for the Arduino with GSM shield that was stable enough to guarantee continuous communication. As we were exploring for the first time both of these endeavors, many setbacks occurred from limited memory to harder to figure out
A P P E N D I X B . A P P E N D I X 2
instability issues. In the end, however, we managed to get both parts fully operational without any known bugs and will proceed to present the final outcome.
GSM communication
Let’s start with the client program that we run on the Arduino master with the GSM/GPRS shield. The one we used for this work was the TinySine SIM 900 Arduino shield[162].
This shield comes with several features being the most important the SIM card holder, the GSM antenna and the SIM 900 modem[163]. Inserting an SD SIM card on the card holder and installing the shield on top of the Arduino provides the opportunity to connect this device to the internet. Communication between the Arduino and the SIM 900 modem is achieved via software serial by the for digital I/O pins 0 and 1 that are the RX (receiver) and TX (transaction) lines respectively. These are the same lines used by the Arduino USB bootloader.
For this purpose, every step of the process must be programmed at a very low level, since there are very few libraries we can use that actually work as intended for our setup.
The type of instructions that a SIM card accepts to connect it’s network are in the form of GSM AT commands which is the abbreviation for ATtention commands. Another reason for this designation is that every command starts withAT. This a widely used technology for many types of the older dial-up internet modems all the way to modern mobile phones GSM modems. However, even though there are some similarities across all modems AT commands, most of them are modem specific. These set of commands can be used not only for establishing an internet connection but can also be used to send SMS and start phone calls. In this work we obviously didn’t use any of those types of communication.
Having stated the complexity of establishing all the communications needed for this work, the actual number of lines of code is not that extensive. The complexity in this case comes from choosing the exact right commands, at the exact timing and order, so that the communication doesn’t stop, fail or bug out. The initialization of the modem as well as the attachment to the GSM network looks like this in our Arduino code (this is a condensed representation as we will not go into detail of every module used, full version isC.1.1):
1 #include "SIM900.h"
2 #include <SoftwareSerial.h>
3 #include "inetGSM.h"
4
5 static char at1[3]="at";
6 static char at2[9]="at+cpin?";
7 static char at3[13]="at+cpin=3220";
8 static char at4[8]="at+csq";
9 static char at5[9]="at+creg?";
10 static char at6[10]="at+cgatt?";
11 static char at7[55]="at+cstt=\"net2.vodafone.pt\",\"vodafone\",\"vodafone\"";
12 static char at8[9]="at+ciicr";
134
B . 6 . S E RV E R C O M M U N I C AT I O N V I A G S M / G P R S
13 static char at9[9]="at+cifsr";
14
15 void setup(){
16 if (gsm.begin(9600))
17 Serial.println(F("\nstatus=READY"));
18 else Serial.println(F("\nstatus=IDLE"));
19 gsmcommand(at1,true);
20 gsmcommand(at2,true);
21 gsmcommand(at3,true);
22 gsmcommand(at4,false);
23 gsmcommand(at5,false);
24 gsmcommand(at6,false);
25 gsmcommand(at7,false);
26 gsmcommand(at8,false);
27 gsmcommand(at9,false);
28 gsm.WhileSimpleRead();
29 }
where we import some libraries that give us some assess to some functions likegsm.begin
,→(). Basically, in this part of the setup we want to follow a recipe to get our device logged in into the GSM network. We need to: unlock the sim via PIN, than ask for registration in the network, than provide our carrier APN (access point name) information, in this case we have SIM from Vodafone, than estabilish the GPRS connection and get a local IP for our device. Note, that some commands are actual instructions and other act like queries of the status of the connection. Regardless, we have found that all of them were actually needed for proper setup. Note that we didn’t useATstrings directly onto the gsm commands. This was done for the purpose of memory optimization, where we define the strings as static variables in order to save some dynamic memory.
After being registered into the network we can than start the processes that will give us assess to the internet. Very briefly, we can say that the steps needed for this are to first establish a TCP/IP connection with the server IP address and then we need to do HTTP interactions in the forms ofHTTP getandHTTP post. Other solutions could have also been possible if the Arduino IP address was public and not local IP inside the Vodafone network. This would allow us to program a server on the Arduino that would wait for requests. As that is not possible in this case our Arduino must request periodically information on the measurement status from a remote server. Our code for the communication with the server and slave looked like this, with some functions being omitted for easier reading (fromC.1.1):
1 #include "SIM900.h"
2 #include <SoftwareSerial.h>
3 #include "inetGSM.h"
4
5 InetGSM inet;
6 char msg[200];
7 char resp[16];
A P P E N D I X B . A P P E N D I X 2
8
9 static char server[15]="193.136.125.67";
10 static char path1[12]="/acq/temp1/";
11 static char path2[13]="/acqt/temp1/";
12
13 void loop() {
14 httpget(path1);
15 Serial.println(F("resp"));
16 Serial.println(resp[0]);
17 if(resp[0]==’N’){
18 delay(1000);
19 Serial.println(F("No"));
20 delay(200);
21 }
22 else if(resp[0]==’G’){
23 delay(1000);
24 httpget(path2);
25 sample_time=atol(resp)*1000*60;
26 Serial.print(F("Sample time: "));
27 Serial.println(sample_time);
28 delay(4000);
29 Wire.beginTransmission(9); // transmit to device #9
30 Wire.write("m"); // sends
31 Wire.endTransmission(); // stop transmitting
32 delay(200);
33 Wire.beginTransmission(9); // transmit to device #9
34 Wire.write(resp); // sends
35 Wire.endTransmission(); // stop transmitting
36 httppost();
37 delay(sample_time);
38 httppostt();
39 }
40 else{
41 delay(1000);
42 delay(200);
43 }
44 delay(60000);
45 }
where we define the main loop for our Arduino master code. Note that in this snippet of the code there are differenthttpget()andhttppost()but not any function for the TCP/IP connection start. The reason for this is that the TCP/IP is open and close for everyhttpget
,→()orhttppost()inside their corresponding functions (that are omitted). Nevertheless, we provide to the Arduino both the IP address of the public server193.136.125.67and the paths for the measurement information/acq/temp1/and/acqt/temp1/.
The Arduino starts by doing anhttpget(path1)to get the information from the server if it is supposed to start a measurement or not. The possible responses areGoorNo. If not, it basically waits for a little over 1 minute to try again. This was done for the purpose of
136
B . 6 . S E RV E R C O M M U N I C AT I O N V I A G S M / G P R S
not overloading both Arduino and the network with constant requests as well as to save some mobile data. The downside is that we can wait up to 1 minute for the measurement procedure to start which is completely acceptable for our purposes. If the response from the server is affirmative then it needs to do anhttpget(path2)to obtain the measuring time. Once this TCP/IP connection closes with response obtained the information is sent via I2C to the slave as we discussed above. When the measurement starts the Arduino sends anhttppost()changing the server side measurement status toStarted. In the end of the sampling time it sends anotherhttppost()changing the server side measurement status toFinished.
Server setup
Let’s now make an overview of how we set up the server. When looking into this part of the project, we opted to build this server to be a public IP address because, as we explained above, the Arduino couldn’t be addressed specifically inside its network. This meant that all kind of connections must be done the other way around, with the Arduino making requests and the server idle waiting for them. We achieved this by contacting the IT department having them register our DNS -http://protautoeuropa.pub.df.fct.
unl.pt/, to a virtual machine (VM) with public IP193.136.125.67. Inside this VM we were then able to build our public server. In this section we will now proceed to explain how we built this server, what functions are implemented and what is the meaning of the information present in the front page (whose clickable url is shown above).
We decided to develop the server and website with the Django framework. Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design. This is a very well documented and popular framework that is rela-tively easy to get into. Adding that to the versatile capabilities of python programming with thousands of libraries for the most diverse applications, this seemed like the best approach.
Django starts with an empty project where you have file that is called manage.py which acts like your main program and when run compiles the rest of the files. You don’t usually tinker with this file, instead there are other files that will build the functionalities and appearance of your server, such as: settings.py(C.2.6),urls.py(C.2.4C.2.5),models.py (C.2.2),views.py(C.2.1) andadmin.pyC.2.3. These are the files that you will need build upon in order to program your server.
InC.2.6we define the global settings for our server. Properties like the server directory, language, timezone, allowed hosts, type of database used, etc. can be defined here. From all these perhaps the most relevant to refer is that our server was built using database type knownsqlite3. This is important because different databases have different requirements and different ways to query the information stored in them. Server databases run in background and are updated with each new interaction with the server. In order to change any type of variable or restructure an existing one you need to migrate the database. This
A P P E N D I X B . A P P E N D I X 2
allows for constant track of the server operation that is persistent even when the server is not running or even if the machine is turned off.
There exist actually two files calledurls.py(C.2.4C.2.5) in two different directories of your server project. Simply put, these files have the task to reference what is shown using what path in the server website. As it might be straightforward to assume you have to program every view of your website to do something specific. urls.pyfiles summarize all the possible views and defines their paths.
The core of our server will be defined inmodels.pyandviews.py. InC.2.2we define the classes of the objects available in the database as well as their properties and operations.
In this file we define two models classesSensorTypeandSensorData. TheSensorTypeclass is defined with the following code:
1 from django.db import models
2 from django.core.validators import validate_comma_separated_integer_list
3
4 class SensorType(models.Model):
5 name = models.CharField(max_length=100)
6 code = models.SlugField(max_length=50)
7 description = models.TextField(blank=True)
8 min_value = models.FloatField()
9 max_value = models.FloatField()\\
10 units = models.CharField(max_length=100)
11 a_type = [(’0’, ’Single’), (’1’, ’Multiple’),]
12 acq_type = models.CharField(max_length=2, choices=a_type, default=0,)
13 acq_time = models.IntegerField(default=0)
14 acq_times = models.CharField(max_length=10000,validators=[
,→validate_comma_separated_integer_list], blank=True, null=True,default=’
,→’)
15 acq = [(’0’, ’Idle’),(’1’, ’Waiting for aquisition’),(’2’, ’Acquiring’),(’3’, ,→ ’Acquisition complete’), ]
16 acquiring = models.CharField(max_length=2, choices=acq, default=0,)
17 val_type = [(’0’, ’Single Value’), (’1’, ’List’),]
18 data_type = models.CharField(max_length=2, choices=val_type, default=0, )
19 log= models.FileField(upload_to=’’,default=’log.txt’)
20 rlog= models.FileField(upload_to=’’,default=’requestlog.txt’)
21
22 def ch_acq(self,val):
23 self.acquiring=val
24
25 def load_acqs(self):
26 values = [int(x) for x in self.acq_times.split(’,’) if x]
27 return values
28
29 def acq_update(self):
30 try:
31 values=self.load_acqs()
32 except(AttributeError):
33 values=[]
138
B . 6 . S E RV E R C O M M U N I C AT I O N V I A G S M / G P R S
34 try:
35 del values[0]
36 self.acq_times=str(values).strip(’[]’)
37 except(IndexError):
38 pass
39 (...)
where we define the properties and operations for objects of this class. This class has the purpose of generating different types of sensors that the Arduino can then refer by their code when delivering data or asking if an acquisition of that sensor is required. This will allow us to create different kinds of inputs for telemetry purposes that the Arduino can then send.
With this in mind, everySensorTypeobject has a full name that shows on the website (name), a code which is an abbreviated name to refer to this sensor and brief descrip-tion field for administrative reasons (descripdescrip-tion). It also has a defined minimum and maximum value (min_valueandmax_value) and units field (units). Then we define the acquisition type (acq_type) has a single measurement or multiple sequential measure-ments. If the first is chosen it has a propertyacq_timewhich is the number of minutes to acquire. If multiple measurements is chosen then we use the propertyacq_timesfor list-ing all the different sampllist-ing times in the sequential set of measurements. Then we define a measurement status propertyacquiringwhich indicates if this sensor is: idle, waiting for and acquisition, acquiring or finished. The output type of this sensor is defined in the propertydata_typehas a single value or an array. Finally, regarding properties there are two.txtlog files which save information on the measurement activity with timestamps (log) and acquisitions requests with timestamps (rlog).
This class also includes three main defined operations:ch_acq,load_acqsandacq_update.
Withch_acqwe can internally change the propertyacquiringto the status parsed.load_acqs is an internal operation that returns the measuring times programmed in the current ac-quisition set. Finally,acq_updateupdates the queue of measurements in the set removing already finished acquisitions.
The second class defined in these models files is theSensorDatawhich uses the follow-ing code:
1 from django.db import models
2 from django.utils.dateparse import parse_datetime
3
4 class SensorData(models.Model):
5 time = models.DateTimeField(auto_now_add=True)
6 type = models.ForeignKey(SensorType, related_name=’data’, on_delete=models.
,→CASCADE)
7 value = models.FloatField(default=0)
8 val_list=models.CharField(max_length=10000,default=list)
9 (...)
A P P E N D I X B . A P P E N D I X 2
Figure B.1: Server homepage developed for this thesis.
where we define the properties of these classes of objects. This is a simpler object whose purpose is solely to carry the data from the sensor. Hence, this object will have an attributetimewhich is the timestamp and a type of data which will be associated with correspondingSensorTypeobject’s type of output. Accordingly, thisSensorDataobject will either use the propertyvalueor the propertyval_list.
Let’s now take a look at the views file (C.2.1). The purpose of this file is define how data will be presented in the website as well as define what actions can be performed.
This is a bigger file so let’s take a look at what the main functions are: home, acquire, acquire_time,plot,sensorandhistory. We won’t go into much detail on the last two as the sensoraction is associated with json output page for mostly debug purposes and thehistory action tracks the values of sensor with time. One important note is that we programmed everything in this server to be as future proof as possible. What this means is that, even though we didn’t have an installed spectrometer in our prototype, we have programmed the server in such a way that is prepared to accept such data and track it with functions likehistory. This was even tested by sending randomly generated data with the Arduino.
Every one of these functions is associated with an url of our website. Some of these actions also work through custom made html templates (C.2.7,C.2.8) and javascript files that we won’t discuss in this main text as they weren’t developed by us. The references for the three javascripts used are: Raphaël 2.1.0 - JavaScript Vector Library,JustGageand jQuery v1.10.2.
Thehomefunction defines what thehomepageof our website looks like with the code:
1 from django.shortcuts import render
2 from .models import SensorType, SensorData
3
140
B . 6 . S E RV E R C O M M U N I C AT I O N V I A G S M / G P R S
4 def home(request):
5 return render(request, ’home/index.html’, {’sensors’: SensorType.objects.
,→filter(data_type=’0’),’sensor_graphs’: SensorType.objects.filter(
,→data_type=’1’),})
where we process a request to access thehomepageand fetch the needed sensor objects from the database. The output is shown in FigureB.1. Where we present the logo of our University as button to this homepage followed by a json button that activates the output from thesensoraction. After that, we show the capabilities of information display of this server by providing several graphs and values. The presented data as it has been said above consists of dummy values sent by the Arduino and is a proof of concept. The meter displays are js scripts that fill up when the page is loaded and the histogram is processed with python’smatplotlibin the functionplotfrom the views file. Using this library within this function we are able to process a standard python plot and convert it as an image HTTP response. The graph generated is stored as an image separately in theurlthat can be accessed by clicking on histogram in the homepage. Clicking on the meter displays will give you access to the recent recorded values for that sensor. Bellow each one of the sensor types we can see their acquisition status.
Perhaps the most important function defined server side is theacquirefunction. In this we define how the server processes client interactions for measurement purposes.
This function uses the following code:
1 from django.shortcuts import render
2 from django.http import HttpResponse, HttpResponseForbidden
3 from .models import SensorType, SensorData
4 from datetime import datetime, timedelta
5
6 def acquire(request, code):
7 if request.method == ’GET’:
8 try:
9 now = datetime.now()
10 rlog=open("requestlog.txt","a+")
11 rlog.write(now.strftime("%m/%d/%Y, %H:%M:%S"))
12 rlog.write(" ")
13 rlog.write("GET")
14 rlog.write("\n")
15 rlog.close()
16 sensor = SensorType.objects.get(code=code)
17 sensor.rlog.name=’requestlog.txt’
18 sensor.save()
19 if(sensor.acquiring==’1’):
20 return HttpResponse(’Go’)
21 else:
22 return HttpResponse(’No’)
23 except SensorType.DoesNotExist:
24 return HttpResponseForbidden
25
A P P E N D I X B . A P P E N D I X 2
26 if request.method == ’POST’:
27 try:
28 sensor = SensorType.objects.get(code=code)
29 value=request.POST[’St’]
30 if(value==’S’ and request.META[’HTTP_USER_AGENT’]==’Arduino/1.0’):
31 sensor.ch_acq(’2’)
32 sensor.acq_update()
33 sensor.save()
34 now = datetime.now()
35 log=open("log.txt","a+")
36 log.write(now.strftime("%m/%d/%Y, %H:%M:%S"))
37 log.write(" ")
38 log.write(sensor.get_acquiring_display())
39 log.write("\n")
40 log.close()
41 sensor.log.name=’log.txt’
42 sensor.save()
43 return HttpResponse(sensor.code + ":" + sensor.acquiring)
44 if(value==’F’ and request.META[’HTTP_USER_AGENT’]==’Arduino/1.0’):
45 if(sensor.acq_type==’1’ and sensor.acq_times!=’’):
46 sensor.ch_acq(’1’)
47 sensor.save()
48 else:
49 sensor.ch_acq(’3’)
50 sensor.save()
51 now = datetime.now()
52 log=open("log.txt","a+")
53 log.write(now.strftime("%m/%d/%Y, %H:%M:%S"))
54 log.write(" ")
55 log.write(sensor.get_acquiring_display())
56 log.write("\n")
57 log.close()
58 return HttpResponse(sensor.code + ":" + sensor.acquiring)
59
60 except SensorType.DoesNotExist:
61 return HttpResponseForbidden
62
63 return HttpResponse()
where we process the incomingHTTP getandHTTP postrequests sent to each sensor. For this reason each generated sensor will have its own acquisition interface url such asthis.
If an HTTP getrequest is sent to this address, the server starts by logging this request with a timestamp inrequestlog.txt. Then it proceeds to check the internal server status for that sensor and sends anHTTPResponseaccordingly. This means that if that sensor is marked withWaiting for acquisition server side the response isGoand otherwise the response isNo.
If the server receives anHTTP postrequest it must check both its headers and the body.
Most importantly on the header front other then the most standard header tags the client 142
B . 6 . S E RV E R C O M M U N I C AT I O N V I A G S M / G P R S
must identify itself asArduino/1.0. Regarding the body of the request the value that the server is expecting to be posted must be in the form ’St=x’ where x will either be ’S’ or ’F’.
The ’St’ here stands for status, while ’S’ means starting and ’F’ finished. When the server receives apostwith the valueSt=Sit knows that the Arduino started the measurement and thus changes in the database of the server the status for that sensor to Acquiring.
Meanwhile, this measurement start is also logged into alog.txtwith a timestamp. When the server receives a post with the valueSt=F it knows that the Arduino finished the measurement and thus changes in the database of the server the status for that sensor to Finished. This also logged into the same log file with a timestamp so we have an accurate history of measurements and their corresponding duration. If any of the header or body message restrictions is not complied by the client the server returns an error response.
Another complementary measurement function is also defined in the views file -acquire_time. As we have seen in the Arduino code, after receiving an authorization to start an acquisition it also must request the desired duration. The server handles this requests with:
1 from django.shortcuts import render
2 from django.http import HttpResponse, HttpResponseForbidden
3 from .models import SensorType, SensorData
4
5 def acquire_time(request, code):
6 if request.method == ’GET’:
7 try:
8 sensor = SensorType.objects.get(code=code)
9 if(sensor.acq_type==’0’ or sensor.acq_times==’’):
10 return HttpResponse(sensor.acq_time)
11 if(sensor.acq_type==’1’):
12 acq_times=sensor.load_acqs()
13 return HttpResponse(acq_times[0])
14
15 except SensorType.DoesNotExist:
16 return HttpResponseForbidden
17
18 if request.method == ’POST’:
19 return HttpResponse()
20
21 return HttpResponse()
where we estabilish a request interface with the server such asthis. In this case the server is only waiting forHTTP getrequests. When these requests happen the server provides that sensor specific next in queue, acquisition time to the client in the form of integer number. Every request that do not follows these rules will result in an error.
Finally, let’s talk about the administration part of this server. Inadmin.pyyou define what this admin page looks like, what functionalities it has available and what info is accessible. Every server in the Django framework is featured with admin page by default.
In this admin section of the server we must provide an user and password that is created