HyperDrive: Update #10

Intended Audience: Marija Bezbradica (FYP Supervisor)

Made progress with:

  • Script to start, stop & restart the HyperDrive application on the server.

I wrote a bash script to simplify the tedious task of starting, stopping and rebooting the application on the server.

The script takes a string as a parameter, the string can either be ‘Start’, ‘Stop’, ‘Restart’. The script contains a case for each event.

When the server is started the output is piped into a log file and the process id is stored in another file. When stopping the process id is read in from that file and the process is killed. The rebooting case is a combination of the first two.
Screen Shot 2016-05-24 at 18.03.57

HyperDrive: Update #9

Intended Audience: Marija Bezbradica (FYP Supervisor)

Made progress with:

  • Writing integration tests for API.
  • Writing unit tests for front end.
  • Implementing E2E testing with protractor.

I left the implementation of most of the testing for the project towards the end so I favoured implementing a number of different types of testing rather than being concerned with a high level of code coverage.

Early in the development process I set up a continuous deployment pipeline to build, test and deploy the project to my development server, after each github commit. I made use of github’s webhooks to trigger a Jenkins job. There was a number of steps involved in the process, Jenkins would:

  • Wipeout the local instance of the project
  • Clone the codebase from github
  • Download all dependencies
  • Run all tests (if successful):
  • Build the project .jar file
  • Connect to development server over ssh
  • Check if an instance of the project was running and stop it.
  • Copy .jar to the server
  • Execute the start script to run the project.

My testing of the API consisted of unit tests for a number of components as well as integration tests for the API endpoints.

I wrote a number of jUnit tests for the Security Utils class. This class is responsible for getting the current user, establishing if they are authenticated as well as checking what roles they are associated with. Each method in this class had a corresponding unit test to establish that they were behaving as expected.
Each of the rest endpoints was given a corresponding test class. These tests differ from the unit tests above because they require a full instance of the application to be running in order for them to execute.

I used the Spring Test Context framework to test almost all of the API methods. The basic workflow of these tests would be to assess the state of a repository, use the MockMVC class to simulate an API call and then assess if the API response was as expect and if the repository had been modified accordingly.

Marking these tests with the @Transactional annotation informed the system to revert the transaction after the test had been completed.
Unit testing for the web application was written in Jasmine and run with KarmaJS. They mock up access to the REST endpoints to enable testing of the UI components.
I used Protractor to automate tests for the front end. Protractor is an end to end testing framework for AngularJs built on top of Selenium Web Driver.

When run it opens a firefox window and simulates click through behaviour. It then evaluates the system behaviour against its expected behaviour.

HyperDrive: Update #8

Intended Audience: Marija Bezbradica (FYP Supervisor)

Made progress with:

  • Server metrics & health checks.
  • Debugging issue with hyper-lapse library.
  • UI improvements to indicate requests are being processed.

Spring Boot Actuator https://github.com/spring-projects/spring-boot/tree/master/spring-boot-actuator is a project built upon Spring Boot which simplifies the addition of monitering and metric gathering functionality.

Using Spring Boot Actuator I was able to expose a Metrics endpoint. Using this endpoint an administrator of the application can view:

  • JVM Metrics: such a memory usage, threads and garbage collection.
  • HTTP Request: amount of requests coming to the API.
  • Ecache & Datasource statistics.

Screen Shot 2016-05-24 at 17.52.13

By evaluating the above metrics a system administrator could make informed decisions about infrastructure requirements and system optimizations tasks.
Spring Boot Actuator also enables us to perform ‘Health checks’. Health checks determine if essential components of the system such as the email server or the database are functioning correctly.

Today I finally got the hyper-lapse functionality to work. Initially I encountered a number of issues while trying to integrate the library into the project. The code I had been using at the start of first semester to test out the library was no longer running. It was logging a number of javascript errors to the console. After quite a bit of digging around I found a solution to the problem in the ‘Issues’ section of the Github repository. It seems like a number of individuals were encountering the same same issue.

Following the advice from the github thread I was able to make a number of adjustments and get the code running.

The video itself takes quite a bit of time to generate, this is due to all the requests that need to be made to Google API. The average time seems to be about 30 seconds. I added a spinning gif within a message dialog at the top of the page, to indicate to the user that their request is being processed.Screen Shot 2016-05-24 at 17.49.17 I made use of the ‘ng-hide’ directive to hide this message once the hyper-lapse is ready to be played.

Screen Shot 2016-05-24 at 17.50.38

 

HyperDrive: Update #7

Intended Audience: Marija Bezbradica (FYP Supervisor)

Made progress with:

  • Overcoming issue resulting in map being drawn without markers.

Recently I’ve struggled with an unusual bug concerned with Google Maps. When a user navigated to the page that ‘journey-map’ page by clicking a hyperlink the map would remain blank. Although if the user refreshed the page the makers would be added to the page after it was reloaded.

I couldn’t make sense of the issue while debugging it. The request for the points was showing up in the server logs and I was even able to log the points to the console. The thing that made the issue so difficult to debug was that every now and then the points would load on the page as expected. While this was a rare occurrence it made me doubt my systematic approach to troubleshooting.

When a request is made for a resource using Angular a promise is used as sort of a placeholder for the data that is to be returned. When the controller was being called I was immediately drawing the map. This resulted in the map being drawn before the promise was resolved resulting in a blank map, yet the points were still being logged to the console.

The solution to this problem was very quick to implement but one that required a thorough understanding of how angular interacts with an external service. I changed the behaviour of the controller to only add the points to the map after the promise had been resolved. This instantly solved the problem that I had spent days troubleshooting.

Screen Shot 2016-05-24 at 18.00.28

HyperDrive: Update #6

Intended Audience: Marija Bezbradica (FYP Supervisor)

Made progress with:

  • Setting up Cordova plugin.
  • Updated mobile UI
  • Transmitting data to the server

In order to get the users current lat, long, altitude and speed I am making use of the Cordova GeoLocation plugin. https://github.com/apache/cordova-plugin-geolocation

It provides a javascript interface to the phones native GPS capabilities. The GitHub page has detailed instructions that I was able to follow in order to integrate it into the mobile application.

I made a number of changes to the sample project I had been working on to bring the UI in line with the projects requirements. Below are some screenshots:

IMG_0941IMG_0938

IMG_0939IMG_0940

I have also implemented services to encapsulate the logic required to interact with the REST API. When the user clicks start driving a request is sent to create a new journey, the server responds with the journey_id which is stored and transmitted with each point as they are recorded. When the user clicks end journey the current journey id is removed from local storage.

HyperDrive: Update #5

Intended Audience: Marija Bezbradica (FYP Supervisor)

Made progress with:

  • Building basic authentication app with Apache Cordova & Ionic framework.
  • Issues communication between mobile app and server. (CORS)

I built a very basic Ionic application which supports token based authentication and authorization.

The app uses ‘angular-mocks’ to simulate a backed. Angular-mocks intercepts http requests and processes them according to the rules you configure.When a user logs into the app a request is made to a back end server with a username and password.I have hardcoded in some behaviour for two types of users ‘user’ & ‘admin’.The regular user has access to the ‘dashboard’ & ‘public’ tabs, while the ‘admin’ user has access to all tabs.

On each state change the app checks if the user is authenticated and authorized. If the user is not authenticated they will be directed back to the login screen. Below are some of the screenshots from the app. On the right the app is making a request for a resource without authentication.

screenshots

After getting the basic mocked functionality working my next step was to make calls to my own API for authentication.

I changed the request being made to add scope, grant type, client_secret, client_id as well as the Basic authorization header.

Screen Shot 2016-04-04 at 19.55.58Screen Shot 2016-03-22 at 12.48.21The above error provided me with quite a bit of difficulty. I was faced with this error when trying to login from the mobile app. I was sending the exactly same request as I was from the web application but I was getting different behaviour. After quite a bit of frustration and research I found out why I was encountering different behaviour.The web application and the API are currently part of the same project, so that when API calls are being made from the front end they are coming from the same origin as the API.

The way I had the project initially configured did not support CORS (Cross Origin Resource Sharing).

If you’re not familiar with CORS the wikipedia page say:

CORS is a mechanism that allows restricted resources (e.g. fonts) on a web page to be requested from another domain outside the domain from which the resource originated. 

Cross-domain” AJAX requests are forbidden by default because of their ability to perform advanced requests (POST, PUT, DELETE and other types of HTTP requests, along with specifying custom HTTP headers) that introduce many cross-site scripting security issues.

I was able to get around this issue by using a filter that allowed CORS requests:

Screen Shot 2016-04-04 at 19.31.24and updating my web security configuration:
Screen Shot 2016-04-04 at 19.33.08

HyperDrive: Update #4

Intended Audience: Marija Bezbradica (FYP Supervisor)

Made progress with:

  • Basic API functionality.
  • Basic UI for registration / login system.
  • Email account validation.

I have completed the basic functionality of the web application. A user can create an account, log into an account and update their information.

As well as some custom CSS the web application uses bootstrap (HTML / CSS framework) with a Material Design theme to make it look a little more professional.

The image below is the initial screen a user will be presented with. At the moment the phone picture and text are just placeholders, which will be updated when the project is completed. Screen Shot 2016-04-04 at 20.34.17

From the nav dropdown ‘account’ a user can choose to either login or register and the appropriate screen will be displayed to the user.
Screen Shot 2016-04-04 at 20.34.30 The registration form fields have some form validation before the ‘register’ button at the bottom becomes active:

  • All fields are required.
  • Email address must have valid syntax.
  • Password must be more then 5 characters long.

There is also an indicator to show users how strong their password is.

Screen Shot 2016-04-04 at 20.34.49

After the user clicks register they are asynchronously sent an email which has a link containing their activation key. The users account does not become activated until the link gets clicked.

I am using a gmail account to send these emails. If I have time left over at the end of the project I might make this look a little more impressive. Here is the email a user receives:

Screen Shot 2016-04-04 at 20.55.55

This screen allows a user to update their information.Screen Shot 2016-04-04 at 20.58.13

Once the user has successfully logged in the ‘Journeys’ option becomes visible in the menu bar. The is the screen that will display a list of the journeys a user has made.

At the moment it holds an embedded google map as well as a basic layout which I shall be improving upon.

Screen Shot 2016-04-04 at 20.36.42

HyperDrive: Update #3

Intended Audience: Marija Bezbradica (FYP Supervisor)

Made progress with:

  • Implementing oauth2 on server.
  • Building basic todo app with Apache Cordova & Ionic framework.
  • Redesigning application entity models.

I’ve been able to implement oauth2 implicit flow (a.k.a. client side flow) in a basic spring boot application. Below is a diagram I found online which helps to visualise the process. In my case the Authorization & REST API tasks are both being handled by the same server.

To speed up the development of the server I used a hard coded username and password. Below you can see a screen shot containing the request parameters. I am using the postman HTTP request client to test the api. oauth2 params

The request also has an ‘Authorization’ header set which contains ‘Basic’ + client_id : clientsecret encoded with base64.

oauth2 request header

When all of the parameters are valid and the username & password match a valid user, an access token and refresh token are generated. These are currently being stored in an h2 database along with a time to live for the tokens. The time to live indicates how long the tokens are valid, before requiring the user to re authenticate with their credentials.

Below is the response received from the server when correct credentials are supplied.

oauth2 responseFor the purpose of this project the server will be using http to reduce complexity. In a real life production environment this type of authentication protocol should always be implemented using HTTPS!

While developing the server has been my priority thus far I have also spend some time getting to grips with mobile development using Apache Cordova and the Ionic framework.

After spending some time reading up on the documentation for both projects I started off by building a basic ‘todo’ application by following a youtube tutorial.

Here are some screenshots I stitched together with photoshop:

todo-screen shots
This project makes use of some basis Ionic components such as:

  • ion-pane
  • ion-header-bar
  • ion-content
  • ion-list
  • ion-item
  • ion-option-button
  • $ionicPopup
  • $ionicListDelegate

When I wasn’t working on the server or mobile application I put some thought into refining the design of the project entities and their relationship with each other.

Here is a diagram of my current entities as they stand. These entities might require some slight changes as the project progresses.Screen Shot 2016-04-04 at 18.59.13

HyperDrive: Update #2

Intended Audience: Marija Bezbradica (FYP Supervisor)

Made progress with:

  • Learning Spring Boot
  • Setting up build / deployment pipeline
  • Setting up mobile development workspace

Learning spring:

Over the last couple of weeks i’ve put a considerable amount of time into learning the basics of spring and spring security.

  • I built a simple login application using basic security.
  • Next is to implement oauth2.

The front end AngularJs code and the API are all in one project at the moment. I’m using the Wro4j maven plugin to resolve the angular dependancies and optimize the resources. I plan on splitting the front and back end into two separate projects. This will give me more flexibility with the tools I use while developing the front end.

Build pipeline:

I’ve spent the last few days setting up the build pipeline, refer to previous post for more details that the process involved. Here’s a basic overview of the system:

  • On each code commit a webhook triggers a Jenkins build.
  • Code is cloned from github, tests are run, if tests pass project jar is built.
  • Jenkins run command to stop HyperDrive API via ssh.
  • Jar file is copied to deployment server.
  • Jenkins starts API via ssh.

Mobile application:

  • Spent time learning the basics of Apache Cordova.
  • Set up development environment.
  • Built basic application using IONIC framework template.
  • Launched application via iOS emulator.

HyperDrive: Setting Up Deployment Server & Continuous Delivery Pipeline

Continuous delivery is a series of practices designed to ensure that code can be rapidly and safely deployed to production by delivering every change to a production-like environment and ensuring business applications and services function as expected through rigorous automated testing.

Throughout the development of the HyperDrive system I plan to use continuous delivery as part of my agile based style of project management.

The first stage of setting up my continuous delivery system was to initially set up the server that will host the HyperDrive API. I am hosting the project on a DigitalOcean VPS running Ubuntu 14.04.3 x64. This is my first experience setting up a VPS and deploying an application.

Heres some of the steps I took for initial setup:

  • Logged in as root
  • Changed root password
  • Created new user danny
  • Granted danny root privileges
  • Created .ssh directory in danny home and added ssh key
  • Disabled root login in ssh daemon
  • Reloaded ssh
  • Install java 8

The next step was to set up my Jenkins server.

I configured as second DigitalOcean VPS running Ubuntu 14.04.3 x64 with similar steps to above, as well as some additional steps:

  • Install Jenkins.
  • Install Maven.
  • Install Git.
  • Create ssh key pair for connecting to deployment server.
  • Setup swap file (to prevent Jenkins crashing when it runs out of memory).
  • Install Jenkins maven plugin.
  • Install Jenkins publish over ssh plugin.

Then I set up my Jenkins build configuration. Heres a list of the steps that are triggered when a change is pushed to GitHub:

  • Checkout code from GitHub.
  • Run Maven clean & package.

If tests pass and build successful:

  • Ssh into deployment server and run:
lsof -t -i tcp:8080 | xargs kill

This kills the process running on port ‘8080’ which shuts down the running instance of the application.

  • Copy .jar file over to deployment server.
  • Run:
#!/bin/sh
cd HyperDrive/target
nohup java -jar *.jar > log.out &

The setup currently meets my needs at the moment. I’m sure there will be some changes made thought the lifecycle of the project. I will update this post with any subsequent changes.