AWS Solutions Architect - Associate Certification

I have been messing around with AWS on little side projects and experiments for about the last year. I did find it quite a daunting experience as there is just so much to learn, do, try, break, struggle, deploy, re-learn and then eventually cheer when everything works.
So I decided to focus a little more and spend the time actually learning how to hopefully do all this properly. I have always found a good way to force myself into the theory and step back from diving directly into implementing solutions is certifications.

I booked the AWS Solutions Architect Associate exam for 4 weeks into the future and set a couple hours a day aside to study.

Main 2 resources I used:

• A Cloud Guru
• I did watched and worked through both the Associate Developer and Associate Architect courses. There is quite a bit of overlap, but still that just meant that I had covered some of the really important bits more than once.
• I really enjoyed Ryan's courses, I did mostly watch on 1.3x speed as it does sometimes feel a little slow and I had set myself a 4 week dead line.
• I would recommend them to anyone starting who's looking to learn about AWS, for practical real life use or certification.
• The discussion section / forum is a great source of tips for the exams as well.

• Cloud Academy
• I only signed up for their 7 day free trial, so did not work through all of their labs. However they have a ton of quiz questions and explanations which is a great way to study / practice specifically for the certification.
• Their summary video was also good and covered some content not covered in depth on a cloud guru.
• They also had some quick easy reading blog posts highlighting important practical information when starting with AWS... listed in Other resources below.

Other Resources:

• http://jayendrapatil.com
• http://cloudacademy.com/blog/aws-bastion-host-nat-instances-vpc-peering-security/
• http://cloudacademy.com/blog/top-13-amazon-virtual-private-cloud-best-practices/
• http://cloudacademy.com/blog/introduction-to-aws-billing-and-cost-management-tools-part-1-of-3/
• http://cloudacademy.com/blog/introduction-to-aws-billing-and-cost-management-tools-part-2-of-3/
• http://cloudacademy.com/blog/introduction-to-aws-billing-and-cost-management-tools-part-3-of-3/
• http://cloudacademy.com/blog/your-first-day-on-aws-10-pitfalls-and-how-to-avoid-them/

The main aws faqs to look at:

• https://aws.amazon.com/vpc/faqs/
• https://aws.amazon.com/s3/faqs/
• https://aws.amazon.com/ec2/faqs/
• https://aws.amazon.com/route53/faqs/

In the end I felt I had personally learnt a lot, not only on the exam topics but in general practical application as well. Main areas were especially with regards to VPCs, Networking and Security. It was well worth the intense 4 week study session.

End Result:

Overall Score: 90%
Topic Level Scoring:
1.0 Designing highly available, cost efficient, fault tolerant, scalable systems : 90%
2.0 Implementation/Deployment: 83%
3.0 Security: 90%
4.0 Troubleshooting: 100%











Next up.. AWS Solutions Architect - Professional

TOGAF 9.1 Notes

I was considering doing the TOGAF 9.1 certification. I have gone over all the content, to at least have a decent idea, what TOGAF is and what it is trying to achieve.
However based on my current position, interests and trends I decided to rather focus my attention elsewhere for the moment.

So just to note down all the resources I found, used and other relevant resources links incase I do have the time or the need to come back to it.

Training course UDEMY (Scott Duffy):
https://www.udemy.com/togaf-enterprise-architect/
https://www.udemy.com/togaf-part2/
I would avoid the Exam Strategy one (https://www.udemy.com/study-togaf) as I found it a little bit of a waste, and he covers a lot of the content in the other 2 courses anyways.

Books / PDFs:
The Actual TOGAF specification (PDF)
TOGAF 9 Foundation Exam Study Guide
TOGAF 9.1 Quick Start Guide for Enterprise Architects (Not sure where I got a PDF from)
TOGAF Cheat Sheet from Scott Duffy's Udemy course.

Other Useful Resources:

http://theopenarch.com/

https://setandbma.wordpress.com/2011/03/31/togaf-foundation-level-certification-another-practice-test/

http://www.bestpracticehelp.com/togafquiz.html

http://class10e.com/TOGAF9/

https://chriseaton.wordpress.com/2009/08/24/togaf-9-certification-multiple-choice-questions/

http://hexopus.com/togaf-study-guide/togaf-9-foundation-quiz/

http://class10e.com/TOGAF9/based-on-togaf-9-which-of-the-following-is-the-best-answer-is-this-the-right-answers-to-choose/

Oracle Workspace Manager - Basic POC with Spring Boot and Flyleaf

Working with a process to update configuration and master data within an enterprise is always a challenging task. While investigating possible solutions on how to have someone change data via a UI, then have those changes tested, signed off and approved before taking it to production. I stumbled on to Oracle Workspace Manager.
According to their developer docs, it seemed fit this use case exactly:

Manage a collection of updates and insertions as a unit before incorporating them into production data
Workspace Manager lets you review changes and roll back undesirable ones before making the changes public. Until you make the changes public, they are invisible to other users of the database, who will access only the regular production data. You can organize the changes in a simple set of workspaces or in a complex workspace hierarchy. A typical example might be a life sciences application in which Workspace Manager supports the discovery and quality assurance (QA) processes by managing a collection of updates before they are merged with the production data.


You could think of Oracle Workspace Manager as light "git"-like db versioning. 
Simply put:

1. You create a workspace (branch)... 
2. You make your changes there.. other people can connect to your workspace .. also make changes... or alternatively create their own branch from yours.
3. You can refresh (fetch / merge) your workspace and resolve any conflicts that arise.
4. Once everything is sorted and all is well with your changes you merge it back into the "LIVE" (origin: develop / master) workspace.

I used an existing Docker image of a oracle standard edition version. Oracle Workspace manager is not available on XE unfortunately. This is a rather large image, and it does take a couple minutes to initialise. The image is available on here on Docker Hub.

Connect database with following setting:

hostname: localhost

port: 1521

sid: xe

service name: xe.oracle.docker

username: system

password: oracle

To connect using sqlplus:

sqlplus system/oracle@//localhost:1521/xe.oracle.docker

To connect on mac os - install instantclient and run from there:

./instantclient_12_1/sqlplus system/oracle@local

To setup the initial DB I tried out Flyway. All quite simple and easy to implement. 
Under resources/db/migration there are a number of sql files that do the initial database setup.

1. create the tables: CODE and CODE_TYPE, 
2. insert initial data 
3. enable versioning on those tables.

When you enable versioning the following happens, the table is renamed and a view is created allowing the "recording" of changes that occur.

Reference: (oracle presentation available here)



So after the initial setup you will seen a number of tables and views:



To try this out...

1. Get the Oracle Docker image
2. Once the DB is started... run the boot DataApplication.
3. Use Postman on the REST resources below

To check the current workspace:
GET: http://localhost:9119/poc/workspace
To select all the information from the code table for the current workspace:
GET: http://localhost:9119/poc/workspace/data/code
Create code in the current workspace:
POST: http://localhost:9119/poc/workspace/data/code
BODY: 
{
"id":2,
"descr": "some code",
"type": 1
}
To change workspace (LIVE is default and available):
PUT: http://localhost:9119/poc/workspace/{workspaceName}
To create a workspace:
POST: http://localhost:9119/poc/workspace/{workspaceName}
To remove a workspace:
DELETE: http://localhost:9119/poc/workspace/{workspaceName}

To merge a workspace:
PUT: http://localhost:9119/poc/workspace/merge/{workspaceName}

These REST resources just wrap some of the functions from the DBMS_WM Package. 
This is maybe just the tip of a iceberg, as there is a ton of functionality available from this package.

All the code is available here.

Re-Inspired

It has been a very long time since I last posted.
 Due to some crazy work pressures and a couple months of working 2 jobs, I just could not find the time nor inspiration to contribute to this blog.

However today a work colleague sent me the following: 52 Technologies in 2016

It's awesome.
It reminded me how much fun I had driving myself to learn, try, demo and blog about new tech all the time. Lets hope life does not get in the way.

Amazon AWS Elastic Beanstalk, Python, Flask and Sci-Stack Docker

This actually took me longer than I'd like to admit to get working, but in the end the solution is quite neat and simple, so it was probably worth it, and hopefully this could save other people some time.

The Amazon Docker file looks like this: 
AWS Elastic Beanstalk Dockerfile - Github

This installs the contents of the root folder requirements.txt before running your Docker file. So for my application the basic "non-sci" packages could be installed simply enough.

Root Folder: requirements.txt:
Then to install the sci related packages... numpy, scipy, pandas, scikit-learn and nltk. I created another requirements.txt in an aws-post-install folder. This is to be run once the Amazon linux OS has been updated and all the required OS dependencies have been installed.

Post Docker requirements.txt:
My custom docker file, that builds ontop of the Amazon image looked as follows:

Docker File:

Next step is to get my docker image to be used directly so that the Elastic Beanstalk app doesn't have to do all the downloads and installs every time should be simple enough according the AWS you tube channel: https://www.youtube.com/watch?v=pLw6MLqwmew

Why Jython when you can microservice with Flask

Over the last little while I have been working on Sibbly it's my little pet project to try summarize, group, filter and target software development information on the web. All 'n all a rather ambitious task, but the worst thing that could happen is that I learn something, so there is really no risk. It is still currently in a very closed beta, only occasionally showing it to fellow work colleagues and getting some input.

After initially starting development for Sibbly on Ubuntu, as I was always planning on deploying on Ubuntu, I had migrated back to windows, and after a couple weeks of work when finally deploying to Ubuntu... Surprise! it obviously did work right off the bat.

The issue I ended up with was, there seems to be a classpath issue between Spring Boot, it's embedded Tomcat instance and Jython. The reason I use Jython is for an awesome library called Pygments.

So after much dismay and checking all the Java alternatives and attempted Pygment ports (jygments, jgments), I started thinking of alternate solutions.
Having recently read: Microservices I decided to look at a way of interacting with Python more indirectly.
This lead me to: Flask
Within a couple minutes thanks to: Awesome Flask Example
I had the following up and running:


What this little bit of Python does is wrap and expose the highlight and guess functionality from Pygments via a RESTful service accepting and producing JSON.

I deploy Sibbly on DigitalOcean
To install Python on my droplet, I followed the process below:

sudo apt-get install python-dev build-essential  
sudo apt-get install zlib1g-dev
sudo apt-get install libssl-dev openssl
sudo apt-get install python-pip
sudo pip install virtualenv
sudo pip install virtualenvwrapper

export WORKON_HOME="$HOME/.virtualenvs"
source /usr/local/bin/virtualenvwrapper.sh

sudo mkdir /opt/python3.4.1
wget http://python.org/ftp/python/3.4.1/Python-3.4.1.tgz
tar xvfz Python-3.4.1.tgz
cd Python-3.4.1
./configure --prefix=/opt/python3.4.1
make  
sudo make install

mkvirtualenv --python /opt/python3.4.1/bin/python3 py-3.4.1

workon py-3.4.1

pip install flask
pip install pygments

Once that was done to run the Flask app:

python app.py & disown

Upgrading Spring 3.x and Hibernate 3.x to Spring Platform 1.0.1 (Spring + hibernate 4.x)

I recent volunteered to upgrade our newest project to the latest version of Spring Platform. What Spring Platform gives you is dependency & plugin management across the whole Spring framework's set of libraries.

Since we had fallen behind a little the upgrade did raise some funnies. Here are the things I ran into:

Maven:
Our pom files were still referencing:
hibernate.jar 
ehcache.jar 
These artefacts don't exit on the latest version, so replaced those with
hibernate-core.jar  and ehcache-core.jar

We also still use the hibernate tools + maven run plugin to reverse engineer our db object.
This I needed to update to a release candidate:

Hibernate:

The code: "Hibernate.createBlob"... no longer exists

replaced with:

On the HibernateTemplate
return types are now List; not element...So needed to add casts for the lists being returned.

import org.hibernate.classic.Session;
replaced with:
import org.hibernate.Session;

Reverse engineer works a little differently...

Assigns Long to numeric...
Added:

Possible Errors:

• Caused by: org.hibernate.service.UnknownUnwrapTypeException: Cannot unwrap to requested type [javax.sql.DataSource]

Add a dependency for c3p0:

And configure the settings in the cfg.xml for it:

• Caused by: java.lang.ClassNotFoundException: org.hibernate.engine.FilterDefinition

Probably still using a reference to hibernate3 factory / bean somewhere, change to hibernate4:
org.springframework.orm.hibernate3.LocalSessionFactoryBean
org.springframework.orm.hibernate3.HibernateTransactionManager

• Caused by: java.lang.ClassNotFoundException: Could not load requested class : org.hibernate.hql.classic.ClassicQueryTranslatorFactory There is minor change in new APIs, so this can be resolved by replacing property value with:

org.hibernate.hql.internal.classic.ClassicQueryTranslatorFactory.

Spring:
Amazingly some of our application context files still referenced the Spring DTD ... replaced with XSD

In Spring configs added for c3p0:

Spring removed the "local"=: so needed to just change that to "ref"=

Spring HibernateDaoSupport no longer has: "releaseSession(session);", which is a good thing so was forced to update the code to work within a transaction.

Possible Errors:

• getFlushMode is not valid without active transaction; nested exception is org.hibernate.HibernateException: getFlushMode is not valid without active transaction

Removed from hibernate properties:

     <prop key="hibernate.current_session_context_class">thread</prop>

Supply a custom strategy for the scoping of the "current"Session. See Section 2.5, “Contextual sessions” for more information about the built-in strategies

• org.springframework.dao.InvalidDataAccessApiUsageException: Write operations are not allowed in read-only mode (FlushMode.MANUAL): Turn your Session into FlushMode.COMMIT/AUTO or remove 'readOnly' marker from transaction definition.

Another option is :

<bean id ="productHibernateTemplate" class="org.springframework.orm.hibernate4.HibernateTemplate">
<property name="sessionFactory" ref="productSessionFactory"/>
<property name="checkWriteOperations" value="false"/>
</bean>

• java.lang.NoClassDefFoundError: javax/servlet/SessionCookieConfig

Servlet version update:

• Then deploying on weblogic javassist: $$_javassist_  cannot be cast to javassist.util.proxy.Proxy

The issue here was that there were different versions of javassist being brought into the ear. I all references removed from all our poms, so that the correct version gets pulled in from from Spring/Hibernate...

and then configured weblogic to prefer our version:

TDD, Hamcrest, Shazamcrest

Recently we have started to try get a more TDD culture started at work, having always believed in thorough testing and decent code coverage it shouldn't have been too hard. However... teaching a old dog new tricks can sometimes require quite a bit of patience. Turns out breaking coding habits formulated of more than a decade of keyboard bashing is harder than it seems.

So with generating an enormous amount of test code, comes the usual task code & test maintenance and reuse.
One of the tools / libraries we have included is Hamcrest, which not only improves the readability of assertion failures, but allows you to create and extend custom matchers, which you can then reuse across multiple test scenarios.

I am not going to go into too much detail on Hamcrest here, where are a bunch of great resources / blogs / tutorials out there.. just a few:
http://www.baeldung.com/hamcrest-collections-arrays
https://weblogs.java.net/blog/johnsmart/archive/2011/12/12/some-useful-new-hamcrest-matchers-collections
http://edgibbs.com/junit-4-with-hamcrest/
http://www.planetgeek.ch/2012/03/07/create-your-own-matcher/

While creating a custom type safe matcher for one of our domain objects, I realised that was insane.. really.. this.getA == that.getA... mmmm no.
So I went searching for something could help and and after a bit, I found: Shazamcrest (bonus points for the name)
What Shazamcrest does is:
Serialize the objects to compare.
Compares them and then on fail throws a ComparisonFailure, which the major IDE's allow you use their build in diff display.

Great... no manual bean compares.

So I add the maven dependency, try it out on our complex domain object....

StackOverflowError.... It was a known limitation at the time. The json provider Shazamcrest was using: 

GSON does not cater for circular reference serialization.

As both Shazamcrest and GSON being opensource, I decided to have a look and see if I could contribute, anything is better that writing a manual bean matcher. After some investigation I found that the guys on the GSON project have created a fix GraphAdapterBuilder, it is just not distributed with the actual library.

So after fork on the Shazamcrest GitHub project, a little bit of code and submitting a pull request:

https://github.com/shazam/shazamcrest/pull/5

The guys on the Shazamcrest project very quickly merged my changes in and published a new version to the maven repo (Thanks for that). 

So be sure to use the 0.8 version if you are struggling with circular references.

Playing with Java 8 - Lambdas, Paths and Files

I needed to read a whole bunch of files recently and instead of just grabbing my old FileUtils.java that I and probably most developers have and then copy from project to project, I decided to have quick look at how else to do it...
Yes, I know there is Commons IO and Google IO, why would I even bother?  They probably do it better, but I wanted to check out the NIO jdk classes and play with lambdas aswell.. and to be honest, I think this actually ended up being a very neat bit of code.

So I had a specific use case:
I wanted to read all the source files from a whole directory tree, line by line.

What this code does, it uses Files.walk to recursively get all the paths from the starting point, it creates a stream, which I then filter to only files that end with the required extension. For each of those files, I use Files.lines to create a stream of Strings, one per line. I trim that, filter out the empty ones and add them to the return collection.
All very concise thanks to the new constructs.

Playing with Java 8 - Lambdas and Concurrency

So Java 8 was released a while back, with a ton of features and changes. All us Java zealots have been waiting for this for ages, all the way back to from when they originally announced all the great features that will be in Java 7, which ended up being pulled.

I have just recently had the time to actually start giving it a real look, I updated my home projects to 8 and I have to say I am generally quite happy with what we got. The java.time API the "mimics" JodaTime is a big improvement, the java.util.stream package is going useful, lambdas are going to change our coding style, which might take a bit of getting used to and with those changes... the quote, "With great power comes great responsibility" rings true, I sense there may be some interesting times in our future, as is quite easy to write some hard to decipher code. As an example debugging the code I wrote below would be "fun"...

The file example is on my Github blog repo

What this example does is simple, run couple threads, do some work concurrently, then wait for them all to complete. I figured while I am playing with Java 8, let me go for it fully...
Here's what I came up with:
Test:
Output:

0 [pool-1-thread-1] Starting: StringInputTask{taskName='Task 1'}
0 [pool-1-thread-5] Starting: StringInputTask{taskName='Task 5'}
0 [pool-1-thread-2] Starting: StringInputTask{taskName='Task 2'}
2 [pool-1-thread-4] Starting: StringInputTask{taskName='Task 4'}
2 [pool-1-thread-3] Starting: StringInputTask{taskName='Task 3'}
3003 [pool-1-thread-5] Done: Task 5
3004 [pool-1-thread-3] Done: Task 3
3003 [pool-1-thread-1] Done: Task 1
3003 [pool-1-thread-4] Done: Task 4
3003 [pool-1-thread-2] Done: Task 2
3007 [Thread-0] WaitingFuturesRunner  - complete... adding results


Some of the useful articles / links I found and read while doing this:

Oracle: Lambda Tutorial
IBM: Java 8 Concurrency
Tomasz Nurkiewicz : Definitive Guide to CompletableFuture

Local Wikipedia with Solr and Spring Data

Continuing with my little AI / Machine Learning research project... I wanted to have a decent sized repo of English text, that was not in a complete mess like a large percentage of data on the internet.  I figured I would try Wikipedia, but what to do with about 40Gb of XML? how do I work / query with all that data. I figured based on recent work implementation where we load something like 200 000 000 records on into a Solr cache, Solr would be the way to go, so the is an example of my basic implementation.

Required for this example:

Wikipedia download (warning it is a 9.9Gb file, extracts to about 42Gb)
Solr
Spring Data (Great Blog / Examples on Spring Data:  Petri Kainulainen's blog)

All the code and unit test for this post is on my blog GitHub Repo

When setting up Solr from scratch, you can have a look at Solr's wiki or documentation, their documentation is pretty good. There is also an example of importing Wikipedia here, I started with that and made some minor modifications.

For this specific example the Solr config needed (/conf):
For this example (and in the below config files),
Solr home: /Development/Solr
Index / Data: /Development/Data/solr_data/wikipedia
Import File: /Development/Data/enwiki-latest-pages-articles.xml

The full import into Solr took about 48 hours on my old 2011 i5 iMac and the index on my current setup is about 52Gb.

Data Config for the import:

Schema:

Solr Config:

The code for this ended up being quite clean, Spring Data - Solr, gives 2 main interfaces SolrIndexService, and SolrCrudRespository, you simply extend / implement these 2, wrap that in a single interface, autowire from a Spring Java context and you good to go.

Repository:

IndexService:

SolrService:


SpringContext:

Next thing for me to look at for sourcing data is Spring Social.