---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------

	HarrysLua - the embedded scripting language for Harry's apps
	(c) 2017 Harald Schlangmann

---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------------
Revisions
---------------------------------------------------------------------------------------------------

	v1.0	22. September 2017		Initial write up focussing on sensor.btle purpose
	v1.1	27. September 2017		Revised samples to latest API, use of gnss, engine, and
									imu instead of gps, obd, acceleration throught the text
	v1.2	2. Octoer 2017			Performance considerations chapter added
	v1.3	10. October 2017		Added section "Is scripting required?"


---------------------------------------------------------------------------------------------------
Introduction
---------------------------------------------------------------------------------------------------

The document gives an introduction into HarrysLua, a scripting language embedded into Harry's apps
and based on Lua 5.3 (see lua.org for reference and programming model).

We have planned to enabled scripting for Harry's apps in several areas, but start with its sensor
framework. Sensor integration has proven to be both complex and broad. "Broad" means there are
hundreds of sensor interfaces out there we can't support using hard coded integration. By making
this area scriptable, we address both complexity and broadness:

	Reduced complexity: the complete path from connection making using the various connection
	technologies (BT, BT LE, WiFi, Mfi), configuration of sensors, and reading / writing proprietary
	protocols is supported by powerful extensions to standard Lua. In addition, standard dialects
	like NMEA for GNSS sensors, or ELM327 for OBD sensors are provided as predefined libraries
	allowing sensor integration scripts as small as 10 lines of code. Scripts are platform
	independent too. An integration scripted for iOS will work on Android and vice versa.

	Allow broadness: scripting allows 3rd parties to provide their own integration - without our
	direct support and independent from app release cycles. Scripts can be written based on examples
	provided and tested "in real life" immediately. In case a 3rd party wants to have a sensor
	integration distributed to all LapTimer users, we can integrate it into release versions too.
	So far, we had to rejects lots of requests by manufacturers and users to support some new
	device. Due to capacity and time limitations, we had to pick the main stream / most common
	devices to support as many users as possible. Scripting will allow much broader support.

In case you are interested in adding scripts, please read this document first. It should give you
an understanding on how scripts are structured and how they can help to support your goals.
Next, head to the Other Sources section and get your hands on the Programming in Lua paperback.
It will give you a full understand of Lua, including embedded scenarios like HarrysLua. Finally,
get you sensor and one of the sample scripts available and customize it to your requirements.
In case you run into any problem, please use the HarrysLua Developer Forum to discuss solutiuons
and request features. We are really excited to make this an universal and often used feature, so
anyone starting the sensor integraton adventure will get our full support!


---------------------------------------------------------------------------------------------------
Availability
---------------------------------------------------------------------------------------------------

HarrysLua will be supported in production versions of Harry's apps starting with major release v22.
We will make this feature available for early adopters in v21.1 Please watch the forum and our
Facebook page for announcements. To allow debugging, we recommend to use Harry's GPS/OBD Buddy
for testing. Other than LapTimer et al, Buddy is logging enabled and will allow you to log both
the framework and your script.

To join the developer program, contact us at Harry@gps-laptimer.de. Developer resources (and
support through HarrysLua Developer Forum in particular) are made available for those joining the
program only.


---------------------------------------------------------------------------------------------------
Is scripting required?
---------------------------------------------------------------------------------------------------

Probably not. Harry's apps supports several levels of sensor individualization:

	Predefined sensors: we try to make genuine sensors used frequently available out of the box.
		Usually connection making on operating system level is all a user needs to do. The
		accessory will be recognized automatically and just work. Predefined sensors are either
		hard coded or scripted (withou exposing this script). Predefined sensors are listed
		on

			http://www.gps-laptimer.de/compatibility/ios

		and

			http://www.gps-laptimer.de/compatibility/android

	Custom sensors: in case you have a plain vanilla GPS sensor feeding data using the NMEA, or
		an OBD sensor using the ELM327 protocol, you can connect it using the app's Expert
		Settings. The only thing to do is entering connection type and parameters. WiFi and BTLE
		connection types are supported for both iOS and Android. For Android, any Bluetooth (SPP)
		sensor speaking the named protocols can be added from Sensor List.

	Scripted sensors: for sensors requiring special connection configurations including custom
		initialization, or those featuring a proprietary protocol, scripting is required. Actually
		scripting is comming in two levels again. For NMEA and ELM327 sensors, a script will
		focus on connection and initialization only. For proprietary protocols, parsing and
		requesting data is scripted individually. See the sections below for the details.


---------------------------------------------------------------------------------------------------
General structure
---------------------------------------------------------------------------------------------------

We start with how a script is created and loaded into Harry's apps. A script files comes with
an extension ".hscrl" and is an XML file on top level. To install the script into one of Harry's
apps on Android, send it to yourself by mail and touch the attachment in e.g. Gmail. Select
Harry's GPS/OBD Buddy from the list of apps shown and you are done. In case you had an older
version of the script on your device already, it is replaced by the new version.

A minimal script file looks something like this:

	<?xml version="1.0" encoding="utf-8"?>
	<scriptdefinition
		xmlns="http://www.gps-laptimer.de"
		xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
		xsi:schemaLocation="http://www.gps-laptimer.de Scripts.xsd">

		<author>Harry</author>
		<version>1.0</version>
		<purpose>sensor.btle</purpose>

		<script><![CDATA[

			...

		]]></script>

	</scriptdefinition>

The purpose defines the "plug-in" point into Harry's. If e.g. a purpose sensor.btle is
chosen, the script will be used to implement a sensor using BT Low Energy technology.

Purposes available or planned so far are:

	sensor.btle
	sensor.wifi
	sensor.bt [TODO]
	sensor.mfi [TODO]
	actioncam [TODO]
	reporting [TODO]
	import [TODO]

Each purpose will define some mandatory and optional functions to be defined by the <script>
tag. The other way around, each purpose will provide a set of functionality beyond pure Lua
that can be used by the script. These sets of functionality are provided by modules.

Purposes and modules form a tight relation:

As a sample, the purpose sensor.btle is a special case of a sensor - supported by functions
predefined in the sensor module. BT LE specialities are supported by the sensor.btle module.
In addition to this module hierarchy, HarrysLua provides a number of global functions and
constants.

Let us have a look into the HarrysLua code itself. For this initial discussion, we use a minimal
script implementing a GPS sensor using a predefined NMEA parser back end. The code shown below
replaces ... in the XML script structure introduced above.

	--	sensor.oninit() is mandatory to define for any sensor purpose
	--	it sets basic parameters and sets up communication

	function sensor.oninit()

		tracecall(connect, "sensor.oninit()")

		--	set sensor parameters
		sensor.channelsets = { gnss } -- one or more channel sets, add from gnss, imu, engine
		sensor.nameprefix = "Custom BTLE" -- shown in Sensor List as "Custom BTLE GNSS"
		sensor.connectiontype = btle -- one of btle, bt, wifi, mfi

		--	set BTLE peripheral name pattern and service / characteristics we are interested in
		sensor.btle.peripheralname = "Some name..." -- regular expression matching sensor name
		sensor.btle.deviceinformation = true -- we assume this sensor provided device information

		--  set our main notifying read charactersitic, we will use the tag returned later
		readcharacteristic = sensor.btle.addcharacteristic(
				"some service uuid", "some characteristiuc uuid", true)

		--	GPS channel specific settings
		-- this is an important one: setting .nmea to true will use a predefined NMEA parser
		sensor.gnss.nmea = true

		tracereturn(connect, "sensor.oninit()")

	end

	-- sensor.btle.onvaluechanged () is mandatory for btle sensor
	-- we redirect input from our read characteristic directly to the predefined sensor.bytesread ()
	-- function, which in turn will pass data to the predefined NMEA parser
	-- (see sensor.gnss.nmea = true above)

	function sensor.btle.onvaluechanged (characteristic, value)

		tracecall(btle, "sensor.btle.onvaluechanged("
				.. characteristic .. ", " .. tracebytes(value) .. ")")

		if characteristic == readcharacteristic then
			sensor.bytesread(value)
		end

		tracereturn(btle, "sensor.btle.onvaluechanged()")

	end

See comments in the code to understand what is actually done. Ignore the trace* statements for
now, they are optional and covered in the section Debugging below.

When a script with sensor purpose is loaded, a function sensor.oninit () is called. This function
is mandatory to define. It describes primary properties of the sensor like which channelsets are
actually supported, the wording in the app's front end (see sensor.nameprefix), and connection
technology specific parameters.

In our BT LE sample, the sensor will go through a discovery phase after returning from
sensor.oninit (). The app is searching for BT LE devices around. To recognize devices, the device
name is used. Valid names are defined by sensor.btle.peripheralname as regular expressions.
If e.g. every actual sensor is coming with a name "FOO" followed by a 3 digit serial number made up
from digits, it could be "FOO[0-9][0-9][0-9]". Once such a sensor (in BT LE a "peripheral") is
discovered, the app will check if the services/characteristics declared using one or more call to
addcharacteristic () are available. Characteristics defined as notifying will automatically be
subscribed to, the sensor is going to state "connected".

In our case, we will see bytes coming in through the read characteristic defined after a few
seconds. Assuming the sensor sends data conforming to the NMEA standard, data is simply dispatched
to sensor.bytesread() here. While this function will often be overridden when reading proprietary
protocols, the default implementation will check if sensor.gnss.nmea (or, for engine channelsets,
sensor.engine.elm327) is true and feed data into that predefined parser. The parser in turn will
queue data into the apps engine once they are recognized.

So if your sensor is using a standard protocol, things are really easy and we are done here. :-)


---------------------------------------------------------------------------------------------------
Custom protocols
---------------------------------------------------------------------------------------------------

So let us have a look into how things work for a custom protocol instead of NMEA. The script
implementing a custom protocol will look like the above one except is sets

	sensor.gnss.nmea = false

in sensor.oninit ().

As the default implementation of sensor.bytesread(value) has no clue how to interpret "value"
with a NMEA parser, we define our own version. Let us assume the sensor is sending (oversimplified)
messages structured like:

	seconds since 1/1/1970 (UTC):	4 bytes, unsigned integer
	longitude:						4 bytes, longitude
	latitude:						4 bytes, latitude
	speed:							1 byte, unsigned integer
	accuracy:						1 byte, unsigned integer

i.e. with 14 bytes overall length each.

	function sensor.bytesread(message)

		if #message == 14 then

			-- create a new and empty set of channels
			gpschannelset = {}

			-- extract data from message and fill gpschannelset table
			gpschannelset.seconds = string.unpack("I4", message, 1)+2082844800
			gpschannelset.longitude = string.unpack("f", message, 5)
			gpschannelset.latitude = string.unpack("f", message, 9)
			gpschannelset.speed = string.unpack("I1", message 13)
			gpschannelset.accuracy = string.unpack("I1", message, 14)

			-- pass result to app
			sensor.queuechannelset(gpschannelset, gnss)

		else

			-- signal app we have received an invalid fix
			sensor.queuechannelset(nil, gnss)

		end

	end


---------------------------------------------------------------------------------------------------
Other important call backs
---------------------------------------------------------------------------------------------------

You may want to initialize your sensor during connection. For situations like this, just define
a function sensor.onconnect(). It will be called after sensor.oninit () and after the sensor has
been connected on technial level. In case you send data to the sensor from here, it is a good idea
to keep the current state to interpret messages correctly

	local initialized = false

	function sensor.onconnect ()

		initialized = false

		-- set up writecharactersitic using addcharacteristic(., ., false) in sensor.oninit ()
		sensors.btle.writevalue(writecharactersitic, "some data")

	end

	function sensor.bytesread(message)

		if initialized then

			... code as before

		else

			... interpret reply from writecharacteristic
			initialized = true

		end

	end

Other optional callbacks are listed in HarrysLua Reference.


---------------------------------------------------------------------------------------------------
Platform limitations
---------------------------------------------------------------------------------------------------

Scripts written in HarrysLua following instructions in this document are portable and will work
for both Android and iOS. There are two limitations worth mentioning:

1) While serial Bluetooth connections (BT SPP) are "open" in Android, this type of connection is
made available for iOS through Apple's Mfi (Made for iPhone) program. The only difference for
the scripts is the purpose (sensor.bt vs. sensor.mfi) and the connection settings in
sensor.oninit (). To allow Harry's apps to access a Mfi device, it needs to be white listed with
Apple's Mfi team. This means for iOS adoptions, there are two approaches: a) White list our apps
(starting with Buddy) for your accessory so the app is allowed to access the accessory. We will
need to release a new app version for this but can do this as part of one or the frequent bug fix
release. Once the app is white listed for the accessory protocol used, you can implement and test
scripts with purpose sensor.mfi. b) Change the firmware of a testing accessory to one of the well
known protocols. Use this protocol setting for implementation and testing of the script. Once you
are ready, do the white listing for production accessories.

2) Scripting available to users is not allowed for iOS. Apps offering such a feature will be
rejected during reviews. This means we do not support user level access for iOS. Instead, the script
is developed by a 3rd party and made available by us and as part of an app release only. Going
this path, the script is an implementation detail that is allowed and used commonly in games.
Please contact us to get support for testing on iOS front up, or simply utilize an Android
device for testing.

In case you have questions on this, please use the HarrysLua Developer Forum.


---------------------------------------------------------------------------------------------------
Debugging
---------------------------------------------------------------------------------------------------

For debugging, we use the same tracing capability LapTimer uses throughout the code. For selected
apps (Harry's GPS/OBD Buddy in general, and all alpha / beta versions of LapTimer), this tracing
capability is enabled and can be used. To allow traces of program flows, tracing needs to be turned
on and the trace classes one is interested in need to be selected. Our apps and the scripts use
the same set of trace classes although we provide only those that make sense as parameters to
select from. Before going into the details, please have a look into the Tracing FAQ available
from the Other Resources section below.

To add tracing output to your scripts, HarrysLua provides three global functions:

	tracecall(), tracereturn(), and trace()

Each of theses functions require two parameters: the trace class and the message to add.

	trace(gnss, "sensor received data: " ..  data)

will add the line "sensor received data: <content of data>" as a seperate line to the trace log.
tracecall() and tracereturn() always come as pairs and build an group in the tracing output. They
are typically used as follows:

	-- some function foogpsfcnt defined in a script
	function foogpsfcnt (argument)

		tracecall(gnss, 'foogpsfctn(argument:' .. argument .. ')')

		-- some function implementation
		result = ...

		-- some other trace output
		trace(gnss, 'some other stuff')

		tracereturn(gnss, 'foogpsfctn()', result)

		return result

	end

The output of the above code will be something like (provided tracing is enabled for the GPS
Sensors trace class):

	GPS         0176292887@main  D 01   --> call to foogpsfctn(argument: <content of argument>)
	GPS         0176292887@main  D 02     --> some other stuff
	GPS         0176292887@main  D 01   --> foogpsfctn() returns ...

In case the GPS Sensors trace class is not enabled in the app (Expert Settings -> Trace Classes),
no output will be generated. Details on trace classes and the above functions can be found in
HarrysLuaReference.

Result handling for tracereturn is optional. In case a function does not return values, the
function is called with just two arguments.

Our apps will check a proper pairing of tracecall() and tracereturn() calls. The function names
need to match up to the opening brace or the argument list for both the call and the return. The
"call to" prefix and "returns" postfix are added automatically, and anything between call and return
will get an increased indent depth (see 02 above).

Our apps follow some conventions when using augmenting code with trace statements. We appreciate
your efforts to follow this conventions to receive consitent output. Once you will start with
tracing, you will see that our apps are heavily instrumented with trace statements and those in
our app and framework code will be mixed with trace statements you add.

When tracing arguments, add the argument name "argument:" and the argument value. This allows
easier reading. Functions not issuing any other output than call and return should use a single
trace statement like

	function foognssfcnt (argument)

		trace(gnss, 'foognssfctn(argument:' .. argument .. ')')

		-- some function implementation

	end

Addition of your own trace classes is not supported. For quick testing, use the adhoc trace class.
It will always output a trace (i.e. there is no trace class that needs to be enabled, it is always
active) and is convenient to use. Please do not forget to either remove the whole call once you do
not need the output any more, or replace adhoc by one of the predefined trace classes before
finalizing the script. The adhoc trace class must not be used in production versions of your
scripts.


---------------------------------------------------------------------------------------------------
Performance considerations
---------------------------------------------------------------------------------------------------

A script based implementation is a lot slower than a hard coded integration using compiled code.
Lua is considered a fast scripting language, but profiling shows it is still slower by a magnitude
when compared e.g. to Objective-C like used in Harry's app on iOS. On the other hand, today's
smartphones are quite fast and will compensate the impact of a scripted portion of code. For a
modern device, any sensor processing up to 25 Hz should not create a relevant slow down. When
integrating 50 or 100 Hz devices however, I'd recommend to monitor the load using either platform
tools or by eableding the Expert Display Mode available in Harry's apps. It will show the
current load in the bottom left of all Timer Views.

In case you run into limits, please contact the Developer Forum to discuss if either additional
native HarrysLua APIs can e introduced, or if the script should be considered a prototype and
go into a hard coded sensor implementation.


---------------------------------------------------------------------------------------------------
Other Resources
---------------------------------------------------------------------------------------------------

Programming in Lua, Fourth Edition: available on all Amazon stores, a great introduction into Lua
https://www.amazon.com/Programming-Lua-Fourth-Roberto-Ierusalimschy/dp/8590379868

HarrysLua Reference: settings, functions and callback reference for HarrysLua. This reference
is available for Developer Program members through the HarrysLua Developer Forum (see below).

Official Lua Site: provids documentation, sources etc: https://www.lua.org

Tracing / Logging Harry's Apps FAQ: http://forum.gps-laptimer.de/viewtopic.php?f=39&t=1933

HarrysLua Developer Forum: http://forum.gps-laptimer.de/viewforum.php?f=47 In case you can't access
the forum, you have not yet contacted us to participate in our developer program.