tag:blogger.com,1999:blog-29767118548922362812024-03-18T14:00:40.353+02:00Zen in the art of ITKnow more...Brianhttp://www.blogger.com/profile/09252223334818664049noreply@blogger.comBlogger120125tag:blogger.com,1999:blog-2976711854892236281.post-29233012976185521142019-06-14T00:01:00.000+02:002019-06-14T00:01:50.379+02:00AWS Certified Solutions Architect - ProfessionalSo I recently needed to re-certify my AWS Certified Solutions Architect - Professional certification. I tried to keep track of everything I did, found helpful or what wasted my time. Hopefully this can aid someone looking at doing this certification, there is so much content to cover.<br />
I spent roughly 2 months prepping for this certification, I did take a week break in the middle, and did skip a couple days here and there.<br />
<br />
Roughly, I did about 100 hours of video content courses and YouTube, 10 practice exams and whole a bunch of reading.<br />
<br />
<span style="font-weight: bold;"><u>Courses & Video Content</u></span><br />
<span style="font-weight: bold;"><u><br /></u></span><b>A Cloud Guru:</b><br />
I have always been a fan of ACG, however I found this course lacking in details to be honest. The actual videos felt like summaries and then you are pointed at other external resources. If you recently did your associate certification, I don't know if you'll get much out of it.<br />
I used this as a a starting point, and it was a good "refresher". The exam simulator as another source of practice, and that is quite good.<br />
<a href="https://acloud.guru/learn/aws-certified-solutions-architect-professional-2019">https://acloud.guru/learn/aws-certified-solutions-architect-professional-2019</a><br />
<br />
<b>Linux Academy:</b><br />
This course covers a lot of the content in some good detail, and gets you to actually try it out in labs. I would recommend everyone signup for this. The instructor and other students are also quite active on their slack channel. I found some of those discussions and exam feedback useful.<br />
<a href="https://linuxacademy.com/amazon-web-services/training/course/name/aws-certified-solutions-architect-professional-2018">https://linuxacademy.com/amazon-web-services/training/course/name/aws-certified-solutions-architect-professional-2018</a><br />
<br />
<b>Exam Readiness: AWS Certified Solutions Architect - Professional (Digital):</b><span style="background-color: #fafafa; caret-color: rgb(84, 91, 100);"><span style="color: #545b64; font-family: inherit;"><br /></span></span><a href="https://www.aws.training/transcript/curriculumplayer?transcriptId=ui4o3L6JBUuG_Zx7hkNvwA2">https://www.aws.training/transcript/curriculumplayer?transcriptId=ui4o3L6JBUuG_Zx7hkNvwA2</a><br />
Was a complete waste of 3 hours of my life.<br />
<br />
<b>You Tube / Re:Invent:</b><br />
Search YouTube for AWS Re:invent deep dive or 300/400 level for any topic you need to cover.<br />
Here are my playlists:<br />
<span style="color: #1a1a1a;"><span style="caret-color: rgb(26, 26, 26);"><a href="https://www.youtube.com/playlist?list=PL9ws_90e5R3wemdD6VLD1qlWEAJytbtu7">Level 400s - Watch all of these</a></span></span><br />
<span style="color: #1a1a1a;"><span style="caret-color: rgb(26, 26, 26);"><a href="https://www.youtube.com/playlist?list=PL9ws_90e5R3yO0gAxCqHWeJBPGWNMWGey">Level 200-300s - Watch what required</a></span></span><br />
<br />
<span style="font-weight: bold;">This is my Architecture:</span><br />
I generally enjoy this format, quick, real-life examples of working architectures.<br />
<a href="https://aws.amazon.com/this-is-my-architecture/">https://aws.amazon.com/this-is-my-architecture/</a><br />
<br />
<br />
<b><u>Notes</u></b><br />
<b><u><br /></u></b>I didn't read a single FAQ, I personally don't like the format. I rather used the aws-cheat-sheets from tutorialdojo.<br />
<a href="https://tutorialsdojo.com/aws-cheat-sheets/">https://tutorialsdojo.com/aws-cheat-sheets/</a><br />
<br />
<br />
<b><u>Practice Exams</u></b><br />
<b><u><br /></u>ACG, Linux Academy:</b><br />
Both had 1 test each, pretty good, would recommend everyone do them. I just found their explanations only focused on the correct answer, where actually identifying incorrect answers is just as important.<br />
<a href="https://acloud.guru/learn/aws-certified-solutions-architect-professional-2019">https://acloud.guru/learn/aws-certified-solutions-architect-professional-2019</a><br />
<a href="https://linuxacademy.com/amazon-web-services/training/course/name/aws-certified-solutions-architect-professional-2018">https://linuxacademy.com/amazon-web-services/training/course/name/aws-certified-solutions-architect-professional-2018</a><br />
<div>
<br /></div>
<b>Whizlabs:</b><br />
These have a bad rep amongst other people taking the test, and I would agree to some extent, I don't know if they added that much value, except for the practice in reading / dissecting questions at speed, and their explanations on some concepts were useful.<br />
<a href="https://www.udemy.com/aws-certified-solutions-architect-professional-whizlabs/">https://www.udemy.com/aws-certified-solutions-architect-professional-whizlabs/</a><br />
<br />
<b>Jon Bonso / Tutorials Dojo:</b><br />
These were good, I would say a little easier than my actual exam, but again great to practice, great to review explanations for correct and incorrect answers<br />
<a href="https://www.udemy.com/aws-solutions-architect-professional-practice-exams-2018/">https://www.udemy.com/aws-solutions-architect-professional-practice-exams-2018/</a><br />
<br />
<b><u>General Reading</u></b><br />
<ul>
<li><a href="https://d1.awsstatic.com/whitepapers/Security/DDoS_White_Paper.pdf">AWS Best Practices for DDoS Resiliency</a></li>
<li><a href="https://d0.awsstatic.com/whitepapers/aws-kms-best-practices.pdf">AWS KMS Best Practices</a></li>
<li><a href="https://d1.awsstatic.com/whitepapers/aws_cloud_adoption_framework.pdf">AWS Cloud Adoption Framework</a></li>
<li><a href="https://d1.awsstatic.com/whitepapers/AWS%20Storage%20Services%20Whitepaper-v9.pdf">AWS Storage Services Overview</a></li>
<li><a href="https://d0.awsstatic.com/whitepapers/architecture/AWS-Cost-Optimization-Pillar.pdf">AWS Cost Optimization Pillar</a></li>
<li><a href="https://d1.awsstatic.com/whitepapers/aws-amazon-vpc-connectivity-options.pdf">AWS VPC Connectivity Options</a></li>
<li><a href="https://d0.awsstatic.com/whitepapers/Security/Secure_content_delivery_with_CloudFront_whitepaper.pdf">Secure Content Delivery with CloudFront</a></li>
<li><a href="https://d1.awsstatic.com/whitepapers/Storage/Backup_and_Recovery_Approaches_Using_AWS.pdf">Backup & Recovery Approaches Using AWS</a></li>
<li><a href="https://d1.awsstatic.com/whitepapers/DevOps/infrastructure-as-code.pdf">Infrastructure as Code</a></li>
<li><a href="https://d0.awsstatic.com/whitepapers/Storage/An_Overview_of_AWS_Cloud_Data_Migration_Services.pdf">An Overview of AWS Cloud Data Migration Services</a></li>
<li><a href="http://d0.awsstatic.com/whitepapers/aws-migrate-resources-to-new-region.pdf?refid=70138000001adyu">Migrating AWS Resource to a New Region</a></li>
<li><a href="https://d1.awsstatic.com/whitepapers/overview-of-deployment-options-on-aws.pdf">Overview of Deployment Options on AWS</a></li>
<li><a href="https://d1.awsstatic.com/whitepapers/Multi_Tenant_SaaS_Storage_Strategies.pdf">SaaS Storage Strategies</a></li>
<li><a href="https://d1.awsstatic.com/whitepapers/aws-web-hosting-best-practices.pdf">Web Application Hosting in AWS Cloud</a></li>
<li><a href="https://d0.awsstatic.com/whitepapers/extend-your-it-infrastructure-with-amazon-vpc.pdf">Extend your IT infrastructure with vpc</a></li>
</ul>
<br />
<br />
<br />
<b><u>The Exam</u></b><br />
<b><u><br /></u></b>
<b>Topics covered in my 75 questions (as much as I can remember) :</b><br />
<ul>
<li><span style="--darkreader-inline-color: #e8e6e3;">AWS Organizations scenarios - Multi AWS accounts, SCP, SAML </span>IAM <span style="--darkreader-inline-color: #e8e6e3;"> 4+</span></li>
<li>Lambda + Api gateway 4+</li>
<li>AWS Systems manager - Patch, Run command, automation, maintenance: 4+</li>
<li>Amazon Aurora vs RDS vs EC2 hosted: 3+</li>
<li>Elastic Beanstalk: 3+</li>
<li>ECS & Fargate: 3+</li>
<li>Cloudformation - nested stacks, stacksets: 3+</li>
<li>CloudFront - Caching & Lambda@Edge: 3 </li>
<li>SQS: 3</li>
<li>EBS - Provisioned IOPS & GP2: 2+</li>
<li>DynamoDB: 2+</li>
<li>ELB & Autoscaling : 2+</li>
<li>Redshift (one HA / quick recovery scenario) : 2+</li>
<li>AWS CodeDeploy & CodePipeline: 2+</li>
<li>Cloudwatch / CloudWatch Logs: 2</li>
<li>Cloudtrail: 2</li>
<li>Direct Connect & Direct Connect Gateway: 2</li>
<li>VPN & Direct Connect routing BGP preferred: 2 </li>
<li>Data Migration Service: 2</li>
<li>AWS WAF & Shield & Shield Advanced (DDoS) : 2</li>
<li>Transit VPC vs Transit Gateway / VPC Peering: 2</li>
<li>Kinesis - family video, firehose, analytics </li>
<li>NAT Instance / Nat Gateway</li>
<li>Route 53</li>
<li>AWS Config</li>
<li>Application Discovery</li>
<li>Snowball, Snowmobile</li>
<li>AWS Batch</li>
<li>Athena & Quicksight</li>
<li>AWS Rekognition</li>
<li>Trusted Advisor (Business Level Support)</li>
<li>Lex</li>
<li>OpsWorks</li>
<li>Service Catalog</li>
<li>AppStream</li>
<li>VPC Flow Logs</li>
<li>VPC Endpoints - Private DNS, S3</li>
<li>Storage Gateway - File and Tape</li>
<li>X-Ray</li>
<li>Amazon Connect</li>
</ul>
<b><br /></b>
<b>My experience writing:</b><br />
<br />
<span style="--darkreader-inline-color: #e8e6e3;"></span>Going to start by saying, I found it harder and the questions less obvious than any of the practice tests. I have a certification exam technique where I flag questions I am not 100% sure of... not necessarily to go back to, but as a counter... I know if I marked less that 20 out of the 75 questions, I would be pretty sure that I would pass.. and can just complete the exam.<br />
<br />
I started, Question 1: easy... ok nice can relax, I got this...<br />
<br />
I then proceeded to "flag" the next 7 questions due thoughts ranging from "mmm ... unsure" to "I have no clue, wtf?".<br />
<br />
Anyways, that trend continued throughout the exam.. couple correct, a whole bunch of "I'm not sure.". I got to the end with around 40 minutes to spare and realised that I had flagged 39 questions, I was quite disheartened at that point, so I went back and tried to review as much as I could.. I got through about 20 flagged questions when I had 9 minutes left, and I resigned myself to the fact that this was my first attempt and starting thinking what should I go back and learn for when I write again in 2 weeks.<br />
<br />
Amazingly, I saw the words... "Congratulations, you have successfully passed ... bla bla bla."<br />
I had to read that a couple times.. and actually still didn't believe it until I actually got my score report the following day.<br />
I somehow managed to get: 880<br />
<br />
So after receiving my score and re-evaluation my exam, I guess my main piece of advice would be:<br />
<br />
The way the questions / scenarios are structured are purposely meant to be a little unclear. Be sure to pay attention to the key words / synonyms / phrases that the question is actually asking for:<br />
<b>cost, fewest changes, fastest processing time, least ongoing maintenance, most secure, highly available, most scalable, least downtime, fastest recovery time</b>.<b>.. etc</b><br />
Then apply what you have learnt about the services, even if you're not 100% sure on the actually implementation of the scenario, make sure it satisfies the core requirement.<br />
<br />
To anyone reading this and planning to write, Good Luck! This is a tough one.<br />
<br />Brianhttp://www.blogger.com/profile/09252223334818664049noreply@blogger.com122Melbourne VIC, Australia-37.8136276 144.96305759999996-39.415753599999995 142.38127059999997 -36.2115016 147.54484459999995tag:blogger.com,1999:blog-2976711854892236281.post-66841350010808992112017-07-06T09:29:00.000+02:002017-07-06T09:29:06.925+02:00The struggle to stay up to date is real... We all know the software development industry is a non stop innovation avalanche<br />
barreling down a mountain.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2LSS4TvW6hFj8jXocHc9Z3s578xUBUDZnrEUSzyWrZqBqif8gEaOONzrREXTMVX0bAPzd1XgQRwP0CZD7DFUvWN-CcaU5XggO8rdm5eBhP7iVt10v13ghTjN-Lei2_TkqZrSVMNgnoxMd/s1600/DevVsTech.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="429" data-original-width="634" height="432" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2LSS4TvW6hFj8jXocHc9Z3s578xUBUDZnrEUSzyWrZqBqif8gEaOONzrREXTMVX0bAPzd1XgQRwP0CZD7DFUvWN-CcaU5XggO8rdm5eBhP7iVt10v13ghTjN-Lei2_TkqZrSVMNgnoxMd/s640/DevVsTech.png" width="640" /></a></div>
<br />
<br />
<br />
So staying up to date with all the tech you touch is becoming a real struggle. When I started out as a developer many years back, I felt staying on the bleeding edge wasn't too bad.<br />
I mean I needed to know and keep in touch with:<br />
<br />
<ul>
<li>My main language and some of its main frameworks.. Java / Junit / Hibernate </li>
<li>Some html, javascript, applets, JSP tags</li>
<li>Some RDBMS and SQL.. Latest changes on Oracle / SQL Server, how their indexing strategies change between versions etc.</li>
</ul>
<br />
These days however the list of technologies, frameworks, languages and tools has just exploded.<br />
<br />
I have in some way or other worked with, tried, read about, implemented, studied, interested in or affected by the following in the last year. Most of which have shipped so much functionality it's actually mind boggling :<br />
<br />
<ul>
<li>Java</li>
<ul>
<li>Java 7 to 8 </li>
<li>Frameworks: Spring, Spring Cloud, Spring Boot, Hibernate </li>
</ul>
<li>JavaScript</li>
<li>TypeScript</li>
<li>Go</li>
<li>Python</li>
<li>Testing:</li>
<ul>
<li>JUnit, Cucumber, Pact.IO, Selenium, Jasmine, Karma, Protractor, Hamcrest, Shazamcrest, Mockito</li>
</ul>
<li>AngularJS</li>
<li>Angular 2 / 4</li>
<li>Docker</li>
<ul>
<li>Docker, Compose, Swarm</li>
</ul>
<li>AWS</li>
<ul>
<li>Lambda, DynamoDB, Step Functions, S3, Cognito, ECS, EC2, VPC, Cloud Formation</li>
</ul>
<li>Netflix</li>
<li>Elastic Search</li>
<li>LogStash, Kibana</li>
<li>Weblogic</li>
<li>Tomcat</li>
<li>Kubernetes</li>
<li>Kafka</li>
<li>Pivotal Cloud Foundry</li>
<li>Caching</li>
<ul>
<li>Hazelcast, Redis, EHCache</li>
</ul>
<li>Storage</li>
<ul>
<li>Oracle, MongoDB, Prometheus.io, Neo4J, H2, MySQL, InfluxDB</li>
</ul>
<li>Build + CI/CD</li>
<ul>
<li>Jenkins, Bamboo, Concourse</li>
</ul>
<li>Git</li>
<li>Windows Server, MacOS, Linux, iOS</li>
<li>Blockchain</li>
</ul>
<br />
I am pretty sure there are a couple others that I am missing... and not mentioning the architectural styles, patterns, agile practices and the like so prevalent in our industry.<br />
<br />
What sparked this blog post was watching the DockerCon Videos from May...<br />
I would say I use Docker pretty regularly, generally on quite simple use cases for my own development and some work related projects...<br />
But watching the videos below I realized that I had completely missed out on a ton of new functionality I didn't know about...<br />
<br />
The reason I could watch a DockerCon video 11AM on a Wednesday morning is because I am current between countries and therefore between jobs... So I have a week or 2 to kill, and obviously I jump at the opportunity to go check out some of the latest stuff...<br />
<br />
However seeing how many changes there were and that I missed a lot of it...<br />
I think I may need to schedule "tech-cations" every couple months to just gorge on all that is new for a couple days.. or hopefully with the use of public transport in my new city, I can cram in another hour or so of podcasts or videos into my daily schedule.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe width="320" height="266" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/Zq6vkaaxML0/0.jpg" src="https://www.youtube.com/embed/Zq6vkaaxML0?feature=player_embedded" frameborder="0" allowfullscreen></iframe></div>
<br />
<br />
<br />
<br />
<br />Brianhttp://www.blogger.com/profile/09252223334818664049noreply@blogger.com55tag:blogger.com,1999:blog-2976711854892236281.post-64606907695297778252017-05-05T19:40:00.000+02:002017-05-05T19:40:27.141+02:00Some interesting videos from the last ng-conf<div>
With so much content out there it is sometimes hard to find content that is actually worth spending your time on.</div>
<div>
<br /></div>
<div>
ng-conf released 66 videos in their last <a href="https://www.youtube.com/playlist?list=PLOETEcp3DkCoS_2cW205cfRGl-Xp5jw4K">playlist</a></div>
<div>
also someone gathered all the relevant resources for the conference available <a href="https://github.com/angular-pakistan/ng-conf-2017">here</a><br />
<br /></div>
<div>
Here are a couple that I watched and found interesting or informative</div>
<div>
<br /></div>
<div>
Angular & GraphQL ... <a href="http://www.apollodata.com/">Apollo</a> looks awesome, something I definitely want to try that on my next angular app / side project.</div>
<br />
<iframe allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/rb5i8Bs7-k0" width="560">Angular and GraphQL – A modern API for a modern Platform - Uri Goldshtein</iframe>
<br />
<br />
I use the angular-cli and I think it's awesome... but John Papa uses it more.. some nice tips<br />
<br />
<iframe allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/h7eVZg3j_Lk" width="560">I AM ONE WITH ANGULAR, ANGULAR IS ONE WITH ME - JOHN PAPA</iframe>
<br />
<br />
RxJs - as I don't get to develop in Angular on a daily basis, always good to get a refresher and learn new functionality<br />
<br />
<iframe allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/TszoFCFydiM" width="560">RxJS: The Good Parts</iframe>
<br />
<br />
More RxJs...
<br />
<br />
<iframe allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/aZRJOVPMW4k" width="560">Reactive Programming with RxJS A Beginner’s Perspective</iframe>
<br />
<br />
Some component design consideration from the guys on the Material angular team.
<br />
<br />
<iframe allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/0BikjL858OQ" width="560">Thoughtful Component Design</iframe>
Brianhttp://www.blogger.com/profile/09252223334818664049noreply@blogger.com8tag:blogger.com,1999:blog-2976711854892236281.post-9086622269644171712017-04-17T12:33:00.000+02:002017-04-17T12:33:13.417+02:00B.A.S.S - Boot Angular Secure StarterAs this is a post related to a project "starter", I decided this time to rather have most of the details in the README.md on github rather than here on my blog.<br />
<br />
To skip my ramblings: <a href="https://github.com/bdupreez/bass" target="_blank">https://github.com/bdupreez/bass</a><br />
<br />
I have recently done a couple little side projects and proof of concepts, all in Angular 2 (or 4 or whatever someone calls it), all using Spring Boot, all requiring some sort of authentication.<br />
<br />
With both Spring and Angular teams iterating and releasing so quickly I found that by the time I got to a new POC I often had to do a number of updates and some of the projects brought in their own unnecessary complications, so it would take longer than I like to get started with the new POC, and therefore I decided it get back to a barebones implementation.<br />
<br />
Hopefully it proves helpful to someone that requires an quickstart full stack Angular / Spring application.<br />
<br />
It can extended and deployed as required.<br />
<br />
One choice I would like to mention as it might not be to everyone's taste is the idea of having the Angular application as a Spring boot application rather than just deploying the static content to Nginx or Apache, and if you're looking to deploy this in a high volume production environment, you'd probably want to relook at that.<br />
The reason I did this was just as convenience for myself. I liked having the whole app deployed as 3 self contained jars and drive both backend and ui configuration from one spring profile environment variable. Of course as this already does contain some of the Spring cloud dependancies, it would be simple enough to include Service discovery with Eureka and a Config Server, but for quickstarter... running 5 or 6 boot applications for a simple web ui might be overkill :)<br />
<br />
<div>
More details on Github Project <a href="https://github.com/bdupreez/bass">README.md</a></div>
<br />Brianhttp://www.blogger.com/profile/09252223334818664049noreply@blogger.com31tag:blogger.com,1999:blog-2976711854892236281.post-42955809110231017292017-03-14T13:46:00.000+02:002017-03-14T13:46:51.878+02:00Consumer Driven Testing with Pact & Spring BootRecently a colleague of mine stumbled across <a href="https://pact.io/" target="_blank">Pact.io</a>, Our current application had grown to over 50 services and we we're starting to have some integration test failures and a brittle dev / acceptance test environment. So we decided to have a look at ways to try help with this.<br />
<br />
I started out by reading:<br />
<a href="https://docs.pact.io/faq/convinceme.html" target="_blank">https://docs.pact.io/faq/convinceme.html</a><br />
<br />
Then watching:<br />
<div class="MsoNormal">
<o:p></o:p></div>
<a href="https://www.youtube.com/watch?v=-6x6XBDf9sQ&feature=youtu.be">https://www.youtube.com/watch?v=-6x6XBDf9sQ&feature=youtu.be</a><br />
<br />
Those 2 resources convinced me to give it a shot.<br />
<br />
So I set out and created a quick set of Spring boot apps, the GitHub repo <a href="https://github.com/bdupreez/pactdemo" target="_blank">here</a>, to test out the concepts and get everything working.<br />
<br />
To highlight some important bits from the demo.<br />
<br />
<b>Consumer:</b><br />
As Pact is a consumer driven test framework. This is where you define a unit test, that test mocks the http server response and you assert against that.<br />
Once the test is successful it creates a pact json file in the /pacts directory.<br />
<br />
<script src="https://gist.github.com/bdupreez/3e071e8347046171a5f2b9e8fd11b690.js"></script>
So after the "mock" test is run and the pact file has been created. You need to include a maven plugin ...pact... that is then used to publish the content of the pacts/ folder to the pact broker... which is defined in the pom as below.<br />
<br />
<script src="https://gist.github.com/bdupreez/40d642b5931b89f4b9b19483f4259fc8.js"></script>
<b>Producer:</b><br />
<br />
This uses the JUnit integration from Pact.io to download the pacts from the broker and then run against an running service.<br />
<br />
Since this already uses a @RunWith annotation, I could not use the spring boot runner. So to get around that as a before class step, I start the Spring boot application, the pacts then gets run against that running instance... and the boot application gets stopped again after the tests. Depending on your use case I guess it would also be an option to do this with @Before so you get a new service instance started before each pack, but that would slow down the execution tremendously.<br />
<br />
The @State annotation, allows for clients to define a specific state, which the producer can the use to setup additional data / conditions required for the test to run.<br />
<br />
Once the pacts have executed against the service there are reports generated in the target folder.<br />
<br />
<script src="https://gist.github.com/bdupreez/18110e00a84bd6eefaf279a5193db835.js"></script>
<br />
<br />
<br />
<b>Setting up the Pact Broker</b><br />
<br />
1. Grab the public images from Docker Hub.<br />
<blockquote class="tr_bq">
docker pull dius/pact_broker<br />
docker pull postgres</blockquote>
<br />
2. Then setup the Postgres DB<br />
<blockquote class="tr_bq">
docker run --name pactbroker-db -e POSTGRES_PASSWORD=ThePostgresPassword -e POSTGRES_USER=admin -d postgres<br />
docker run -it --link pactbroker-db:postgres --rm postgres psql -h postgres -U admin<br />
CREATE USER pactbrokeruser WITH PASSWORD 'TheUserPassword';<br />
CREATE DATABASE pactbroker WITH OWNER pactbrokeruser;<br />
GRANT ALL PRIVILEGES ON DATABASE pactbroker TO pactbrokeruser;</blockquote>
3. Once the DB is up, run the actual Broker:<br />
<blockquote class="tr_bq">
docker run --name pactbroker --link pactbroker-db:postgres -e PACT_BROKER_DATABASE_USERNAME=pactbrokeruser -e PACT_BROKER_DATABASE_PASSWORD=TheUserPassword -e PACT_BROKER_DATABASE_HOST=postgres -e PACT_BROKER_DATABASE_NAME=pactbroker -d -p 80:80 dius/pact_broker</blockquote>
<br />
<br />
<b>Extra References:</b><br />
<br />
<a href="https://docs.pact.io/documentation/">https://docs.pact.io/documentation/</a><br />
<a href="https://docs.pact.io/documentation/sharings_pacts.html">https://docs.pact.io/documentation/sharings_pacts.html</a><br />
<a href="https://github.com/DiUS/pact-jvm">https://github.com/DiUS/pact-jvm</a><br />
<a href="https://github.com/DiUS/pact-jvm/tree/master/pact-jvm-consumer-junit">https://github.com/DiUS/pact-jvm/tree/master/pact-jvm-consumer-junit</a><br />
<br />
<b>Get the example project</b><br />
<a href="https://github.com/bdupreez/pactdemo">https://github.com/bdupreez/pactdemo</a><br />
<br />
<div style="background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14px; margin-top: 10px; padding: 0px;">
<br /></div>
Brianhttp://www.blogger.com/profile/09252223334818664049noreply@blogger.com22tag:blogger.com,1999:blog-2976711854892236281.post-30954119782974495672017-02-12T08:39:00.001+02:002017-02-12T08:39:16.897+02:00Quick Docker environment cleanupJust a quick reference on my docker usage.<br />
<br />
When working / playing with docker a lot, you quite often end up with a ton of containers and volumes laying around after they are needed...<br />
<br />
Here are a couple tips I use to keep my environment clean.<br />
<br />
<b>On Windows 10</b><br />
If you are using GitBash create a .bashrc and include the commands below:<br />
<blockquote class="tr_bq">
<br />alias docker-rm='docker rm $(docker ps -q -f status=exited)'<br />alias docker-rmi='docker rmi $(docker images -q -f dangling=true)'<br />alias docker-rmv='docker volume rm $(docker volume ls -q -f dangling=true)'</blockquote>
<br />
<b>For MacOS X </b><br />
Add the following to your .bash_profile:<br />
<br />
<blockquote class="tr_bq">
docker-rm(){<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>docker rm $(docker ps -q -f status=exited)<br />}<br />docker-rmv(){<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>docker volume rm $(docker volume ls -q -f dangling=true)<br />}<br />docker-rmi(){<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>docker rmi $(docker images -q -f dangling=true)<br />}<br />docker-update-all() {<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>docker images | awk '(NR>1) && ($2!~/none/) {print $1":"$2}'| xargs -L1 docker pull<br />}</blockquote>
<br />
<br />
Then simply run<b> docker-rm</b>, <b>docker-rmi</b> and <b>docker-rmv</b> to remove all the dangly bits :)<br />
<br />
<b>docker-update-all</b>, is a handy way to get the latest images from docker hub.Brianhttp://www.blogger.com/profile/09252223334818664049noreply@blogger.com20tag:blogger.com,1999:blog-2976711854892236281.post-42784034073332495862016-12-18T15:37:00.001+02:002019-06-16T04:03:03.910+02:00AWS Solutions Architect - Professional Certification<br />
<b>Update:</b><br />
Check out my updated re-certification on the new 2019 exam... <a href="https://www.briandupreez.net/2019/06/aws-certified-solutions-architect.html">here</a><br />
<br />
Let me start by saying, for this certification I studied and prepared more than any certification I have done before and I have done a number of certifications throughout my career; Microsoft, Sun, IBM, Oracle and now AWS. It does feel like there is a mountain of content to get through, and then ensuring you know it at the correct level of detail for the exam.<br />
<br />
I booked the AWS Solutions Architect Professional exam 5 weeks after the associate level one and set a couple hours a day aside to study... During the last 2 weeks however, those couple hours became all my available hours every day, it was all I focused on, including 8+ hours a day on the weekends.<br />
<br />
<b>Main 4 resources I used:</b><br />
<div>
<ul>
<li><b>YouTube</b></li>
<ul>
<li>The re:Invent sessions videos are an essential resource</li>
<li>My major topics playlist: </li>
<b><br /><iframe allowfullscreen="" frameborder="0" height="344" src="https://www.youtube.com/embed/videoseries?list=PL9ws_90e5R3wfmBn0-XPl38T9R4GebQGW" width="425"></iframe></b></ul>
<ul><b><br /></b>
<li>Watch these twice:</li>
</ul>
<ul>
<b><iframe allowfullscreen="" frameborder="0" height="344" src="https://www.youtube.com/embed/videoseries?list=PL9ws_90e5R3zxXxUVRUYdq9_zvCiLvG1A" width="425"></iframe></b>
</ul>
<ul><b><br /></b></ul>
<li><b><a href="https://linuxacademy.com/" target="_blank">Linux Academy </a></b></li>
<ul>
<li>These guys really impressed me, the content in the pro level certificate was excellent, I found it had more depth than A Cloud Guru, well worth the $29 per month subscription.</li>
</ul>
<li><b><a href="https://acloud.guru/" target="_blank">Cloud Guru</a></b></li>
<ul>
<li>Decent structured content that covers the highlights of all the topics defined in the exam blueprint.</li>
<li>The discussion forum is an excellent source of tips and discussions on the practice exam questions.</li>
</ul>
<li><a href="https://www.webassessor.com/wa.do?page=publicHome&branding=AMAZON" target="_blank"><b>The official practice exam</b></a></li>
<ul>
<li>This was a good eye opener, into how hard some of the questions can be. I do feel that some of the questions are badly articulated which makes them almost impossible to answer, however working through every question and researching the answers was invaluable. There were also a couple of those questions on the real exam... word for word.</li>
</ul>
</ul>
</div>
<div>
<br /></div>
<div>
<b>Other Resources (Summary / Study Notes):</b><br />
<ul>
<li><a href="https://www.blogger.com/%C2%A0http://jayendrapatil.com" target="_blank">http://jayendrapatil.com</a></li>
<li><a href="https://blue-clouds.com/category/aws-certified-solutions-architect-professional/" target="_blank">blue-clouds.com - study notes</a></li>
<li><a href="http://blogs.catapultsystems.com/cmoore/archive/2016/01/27/aws-certified-solutions-architect-study-guide-introduction/" target="_blank">Clay Moore - Blog</a></li>
</ul>
<div>
<br /></div>
</div>
<div>
<div style="font-family: Calibri; margin-bottom: 12pt; margin-top: 0pt;">
<b>The main aws faqs to look at:</b></div>
<ul>
<li><span lang="en-GB" style="background-color: white;"><a href="https://aws.amazon.com/s3/faqs/">https://aws.amazon.com/s3/faqs/</a></span></li>
<li><a href="https://aws.amazon.com/vpc/faqs/" style="background-color: white;">https://aws.amazon.com/vpc/faqs/</a></li>
<li><a href="https://aws.amazon.com/ec2/faqs/" target="_blank">https://aws.amazon.com/ec2/faqs/</a></li>
<li><a href="https://aws.amazon.com/route53/faqs/" style="background-color: white;">https://aws.amazon.com/route53/faqs/</a></li>
<li><a href="https://aws.amazon.com/cloudfront/faqs/" target="_blank">https://aws.amazon.com/cloudfront/faqs/</a></li>
<li><a href="https://aws.amazon.com/directconnect/faqs/" target="_blank">https://aws.amazon.com/directconnect/faqs/</a></li>
<li><a href="https://aws.amazon.com/storagegateway/faqs/" target="_blank">https://aws.amazon.com/storagegateway/faqs/</a></li>
</ul>
</div>
<br />
<b>Final thoughts:</b><br />
<b><br /></b>
All the guides and sites recommend reading a ton of white papers, but to be honest, I read only 4 or 5 of them..and I felt that the time spent was a waste. It seems all the content was in all the youtube vids and online courses already.<br />
<br />
I did see a lot of people mention that they were pressed for time during the exam, but I found that I finished the 77 questions with a good hour to spare. To mention though: I did not go back and review any of my answers. I have always found in the past, that I rarely change my initial answer during the review phase, and I was reasonably confident that I had done enough.<br />
<br />
<h2>
<span style="font-size: small;">
End Result:</span></h2>
<div>
<span style="font-family: "helvetica"; font-size: 12px;"><b>Overall Score: 83%</b></span><br />
<br style="font-family: Helvetica; font-size: 12px;" />
<span style="font-family: "helvetica"; font-size: 12px;"><b>Topic Level Scoring:</b></span><br />
<span style="font-family: "helvetica"; font-size: 12px;">1.0 High Availability and Business Continuity: 90%</span><br />
<span style="font-family: "helvetica"; font-size: 12px;">2.0 Costing: 100%</span><br />
<span style="font-family: "helvetica"; font-size: 12px;">3.0 Deployment Management: 85%</span><br />
<span style="font-family: "helvetica"; font-size: 12px;">4.0 Network Design: 57%</span><br />
<span style="font-family: "helvetica"; font-size: 12px;">5.0 Data Storage: 81%</span><br />
<span style="font-family: "helvetica"; font-size: 12px;">6.0 Security: 78%</span><br />
<span style="font-family: "helvetica"; font-size: 12px;">7.0 Scalability & Elasticity: 90%</span><br />
<span style="font-family: "helvetica"; font-size: 12px;">8.0 Cloud Migration & Hybrid Architecture: 85%</span></div>
<div>
<span style="font-family: "helvetica"; font-size: 12px;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhxwcT5PcBw5cVdBGv2NMUBI0XOSVrpl83tlZ08gwVA8KuTku-aFPr9e4bNoq28SsCE5ANpdbRKZKysvU7An-GlIPN4-PMmzmmT-GcIxZrzwAiruZv7Hc5QnxSV_DbN5sysbXUdNtRJS1nM/s1600/Solutions-Architect-Professional.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="130" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhxwcT5PcBw5cVdBGv2NMUBI0XOSVrpl83tlZ08gwVA8KuTku-aFPr9e4bNoq28SsCE5ANpdbRKZKysvU7An-GlIPN4-PMmzmmT-GcIxZrzwAiruZv7Hc5QnxSV_DbN5sysbXUdNtRJS1nM/s320/Solutions-Architect-Professional.png" width="320" /></a></div>
<div>
<span style="font-family: "helvetica"; font-size: 12px;"><br /></span></div>
Brianhttp://www.blogger.com/profile/09252223334818664049noreply@blogger.com759tag:blogger.com,1999:blog-2976711854892236281.post-87024042030916855082016-11-12T12:47:00.003+02:002016-11-12T12:58:56.761+02:00AWS Solutions Architect - Associate CertificationI have been messing around with AWS on little side projects and experiments for about the last year. I did find it quite a daunting experience as there is just so much to learn, do, try, break, struggle, deploy, re-learn and then eventually cheer when everything works.<br />
So I decided to focus a little more and spend the time actually learning how to hopefully do all this properly. I have always found a good way to force myself into the theory and step back from diving directly into implementing solutions is certifications.<br />
<br />
I booked the AWS Solutions Architect Associate exam for 4 weeks into the future and set a couple hours a day aside to study.<br />
<br />
<b>Main 2 resources I used:</b><br />
<br />
<ul>
<li><a href="https://acloud.guru/" target="_blank">A Cloud Guru</a></li>
<ul>
<li>I did watched and worked through both the Associate Developer and Associate Architect courses. There is quite a bit of overlap, but still that just meant that I had covered some of the really important bits more than once.</li>
<li>I really enjoyed Ryan's courses, I did mostly watch on 1.3x speed as it does sometimes feel a little slow and I had set myself a 4 week dead line.</li>
<li>I would recommend them to anyone starting who's looking to learn about AWS, for practical real life use or certification.</li>
<li>The discussion section / forum is a great source of tips for the exams as well.</li>
</ul>
</ul>
<ul>
<li><a href="https://cloudacademy.com/" target="_blank">Cloud Academy</a></li>
<ul>
<li>I only signed up for their 7 day free trial, so did not work through all of their labs. However they have a ton of quiz questions and explanations which is a great way to study / practice specifically for the certification.</li>
<li>Their summary video was also good and covered some content not covered in depth on a cloud guru.</li>
<li>They also had some quick easy reading blog posts highlighting important practical information when starting with AWS... listed in Other resources below.</li>
</ul>
</ul>
<b>Other Resources:</b><br />
<ul>
<li><a href="https://www.blogger.com/%C2%A0http://jayendrapatil.com" target="_blank">http://jayendrapatil.com</a></li>
<li><a href="http://cloudacademy.com/blog/aws-bastion-host-nat-instances-vpc-peering-security/" style="background-color: white; font-family: Calibri;">http://cloudacademy.com/blog/aws-bastion-host-nat-instances-vpc-peering-security/</a></li>
<li><a href="http://cloudacademy.com/blog/top-13-amazon-virtual-private-cloud-best-practices/" style="font-family: Calibri;" target="_blank">http://cloudacademy.com/blog/top-13-amazon-virtual-private-cloud-best-practices/</a></li>
<li><a href="http://cloudacademy.com/blog/introduction-to-aws-billing-and-cost-management-tools-part-1-of-3/" target="_blank">http://cloudacademy.com/blog/introduction-to-aws-billing-and-cost-management-tools-part-1-of-3/</a></li>
<li><a href="http://cloudacademy.com/blog/introduction-to-aws-billing-and-cost-management-tools-part-2-of-3/" target="_blank">http://cloudacademy.com/blog/introduction-to-aws-billing-and-cost-management-tools-part-2-of-3/</a></li>
<li><a href="http://cloudacademy.com/blog/introduction-to-aws-billing-and-cost-management-tools-part-3-of-3/" target="_blank">http://cloudacademy.com/blog/introduction-to-aws-billing-and-cost-management-tools-part-3-of-3/</a></li>
<li><a href="http://cloudacademy.com/blog/your-first-day-on-aws-10-pitfalls-and-how-to-avoid-them/" target="_blank">http://cloudacademy.com/blog/your-first-day-on-aws-10-pitfalls-and-how-to-avoid-them/</a></li>
</ul>
<br />
<div style="font-family: Calibri; margin-bottom: 12pt; margin-top: 0pt;">
<b>The main aws faqs to look at:</b></div>
<div lang="en-US" style="font-family: Calibri; font-size: 12.0pt; margin: 0in;">
</div>
<ul>
<li><a href="https://aws.amazon.com/vpc/faqs/" style="background-color: white;">https://aws.amazon.com/vpc/faqs/</a></li>
<li><span lang="en-GB" style="background-color: white;"><a href="https://aws.amazon.com/s3/faqs/">https://aws.amazon.com/s3/faqs/</a></span></li>
<li><a href="https://aws.amazon.com/ec2/faqs/" target="_blank">https://aws.amazon.com/ec2/faqs/</a></li>
<li><a href="https://aws.amazon.com/route53/faqs/" style="background-color: white;">https://aws.amazon.com/route53/faqs/</a></li>
</ul>
<br />
In the end I felt I had personally learnt a lot, not only on the exam topics but in general practical application as well. Main areas were especially with regards to VPCs, Networking and Security. It was well worth the intense 4 week study session.<br />
<h3>
<b>End Result:</b></h3>
<b style="font-family: helvetica; font-size: 12px;">Overall Score: 90%</b><br />
<span style="font-family: "helvetica"; font-size: 12px;"><b>Topic Level Scoring:</b></span><br />
<span style="font-family: "helvetica"; font-size: 12px;">1.0 Designing highly available, cost efficient, fault tolerant, scalable systems : 90%</span><br />
<span style="font-family: "helvetica"; font-size: 12px;">2.0 Implementation/Deployment: 83%</span><br />
<span style="font-family: "helvetica"; font-size: 12px;">3.0 Security: 90%</span><br />
<span style="font-family: "helvetica"; font-size: 12px;">4.0 Troubleshooting: 100%</span><br />
<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhb0ffn_Dqbyne_3vpTOyrvQap75XbqXzBETh_G7ApaD6x0lBd4_kzWPwXhyphenhyphenmtBZztTUPcpi2f69jkv-MIi2e4Ggb4bbwXOkgDgHcrIGs9BVzxyAFm34oVE2R67XTdNrMeBQNz5tRpVRbNT/s1600/Solutions-Architect-Associate.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhb0ffn_Dqbyne_3vpTOyrvQap75XbqXzBETh_G7ApaD6x0lBd4_kzWPwXhyphenhyphenmtBZztTUPcpi2f69jkv-MIi2e4Ggb4bbwXOkgDgHcrIGs9BVzxyAFm34oVE2R67XTdNrMeBQNz5tRpVRbNT/s1600/Solutions-Architect-Associate.png" /></a><br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
Next up.. AWS Solutions Architect - ProfessionalBrianhttp://www.blogger.com/profile/09252223334818664049noreply@blogger.com265tag:blogger.com,1999:blog-2976711854892236281.post-74119217583657402032016-10-08T07:33:00.000+02:002016-10-08T07:33:18.284+02:00TOGAF 9.1 NotesI was considering doing the TOGAF 9.1 certification. I have gone over all the content, to at least have a decent idea, what TOGAF is and what it is trying to achieve.<br />
However based on my current position, interests and trends I decided to rather focus my attention elsewhere for the moment.<br />
<br />
So just to note down all the resources I found, used and other relevant resources links incase I do have the time or the need to come back to it.<br />
<br />
Training course UDEMY (Scott Duffy):<br />
<a href="https://www.udemy.com/togaf-enterprise-architect/" target="_blank">https://www.udemy.com/togaf-enterprise-architect/</a><br />
<a href="https://www.udemy.com/togaf-part2/" target="_blank">https://www.udemy.com/togaf-part2/</a><br />
I would avoid the Exam Strategy one (<a href="https://www.udemy.com/study-togaf" target="_blank">https://www.udemy.com/study-togaf</a>) as I found it a little bit of a waste, and he covers a lot of the content in the other 2 courses anyways.<br />
<br />
Books / PDFs:<br />
<a href="https://www.opengroup.org/togaf/" target="_blank">The Actual TOGAF specification</a> (<a href="https://www2.opengroup.org/ogsys/jsp/publications/PublicationDetails.jsp?catalogno=I112" target="_blank">PDF</a>)<br />
<span class="a-size-extra-large" id="ebooksProductTitle" style="box-sizing: border-box; font-weight: normal; line-height: 1.2 !important; overflow-wrap: break-word; text-rendering: optimizelegibility; word-wrap: break-word;"><span style="font-size: small;"><a href="https://www.amazon.com/TOGAF-Foundation-Exam-Study-Guide-ebook/dp/B008UTO2CA" target="_blank">TOGAF 9 Foundation Exam Study Guide</a></span></span><br />
TOGAF 9.1 Quick Start Guide for Enterprise Architects (Not sure where I got a PDF from)<br />
TOGAF Cheat Sheet from Scott Duffy's Udemy course.<br />
<br />
Other Useful Resources:<br />
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<a href="http://theopenarch.com/">http://theopenarch.com/</a></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<a href="https://setandbma.wordpress.com/2011/03/31/togaf-foundation-level-certification-another-practice-test/" style="font-size: 11pt;">https://setandbma.wordpress.com/2011/03/31/togaf-foundation-level-certification-another-practice-test/</a></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<a href="http://www.bestpracticehelp.com/togafquiz.html">http://www.bestpracticehelp.com/togafquiz.html</a></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<a href="http://class10e.com/TOGAF9/">http://class10e.com/TOGAF9/</a></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<a href="https://chriseaton.wordpress.com/2009/08/24/togaf-9-certification-multiple-choice-questions/">https://chriseaton.wordpress.com/2009/08/24/togaf-9-certification-multiple-choice-questions/</a></div>
<div lang="en-GB" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<a href="http://hexopus.com/togaf-study-guide/togaf-9-foundation-quiz/">http://hexopus.com/togaf-study-guide/togaf-9-foundation-quiz/</a></div>
<!--StartFragment-->
<!--EndFragment--><br />
<div lang="en-GB" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<a href="http://class10e.com/TOGAF9/based-on-togaf-9-which-of-the-following-is-the-best-answer-is-this-the-right-answers-to-choose/">http://class10e.com/TOGAF9/based-on-togaf-9-which-of-the-following-is-the-best-answer-is-this-the-right-answers-to-choose/</a></div>
Brianhttp://www.blogger.com/profile/09252223334818664049noreply@blogger.com28tag:blogger.com,1999:blog-2976711854892236281.post-1812932293829556402016-09-11T10:43:00.001+02:002016-09-11T10:58:41.415+02:00Oracle Workspace Manager - Basic POC with Spring Boot and Flyleaf<span style="font-family: inherit;">Working with a process to update configuration and master data within an enterprise is always a challenging task. While investigating possible solutions on how to have someone change data via a UI, then have those changes tested, signed off and approved before taking it to production. I stumbled on to Oracle Workspace Manager.</span><br />
<span style="font-family: inherit;">According to their developer </span><a href="http://docs.oracle.com/cd/B19306_01/appdev.102/b14253/long_intro.htm" style="font-family: inherit;" target="_blank">docs</a><span style="font-family: inherit;">, it seemed fit this use case exactly:</span><br />
<span style="font-family: inherit;"><br /></span>
<i><b><span style="color: #999999;">Manage a collection of updates and insertions as a unit before incorporating them into production data</span></b></i><br />
<i><span style="color: #999999;">Workspace Manager lets you review changes and roll back undesirable ones before making the changes public. Until you make the changes public, they are invisible to other users of the database, who will access only the regular production data. You can organize the changes in a simple set of workspaces or in a complex workspace hierarchy. A typical example might be a life sciences application in which Workspace Manager supports the discovery and quality assurance (QA) processes by managing a collection of updates before they are merged with the production data.</span></i><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">You could think of Oracle Workspace Manager as light "git"-like db versioning. </span><br />
<span style="font-family: inherit;">Simply put:</span><br />
<br />
<ol>
<li><span style="font-family: inherit;">You create a workspace (branch)... </span></li>
<li><span style="font-family: inherit;">You make your changes there.. other people can connect to your workspace .. also make changes... or alternatively create their own branch from yours.</span></li>
<li><span style="font-family: inherit;">You can refresh (fetch / merge) your workspace and resolve any conflicts that arise.</span></li>
<li><span style="font-family: inherit;">Once everything is sorted and all is well with your changes you merge it back into the "LIVE" (origin: develop / master) workspace.</span></li>
</ol>
<br />
<div>
<span style="font-family: inherit;">I used an existing Docker image of a oracle standard edition version. Oracle Workspace manager is not available on XE unfortunately. This is a rather large image, and it does take a couple minutes to initialise. The image is available on here on </span><a href="https://hub.docker.com/r/sath89/oracle-12c/" style="font-family: inherit;" target="_blank">Docker Hub</a>.<br />
<span style="font-family: inherit;"><br /></span>
<br />
<div style="color: #555555; margin: 0in;">
<span style="font-family: inherit;"><b>Connect database with following setting:</b></span></div>
<div style="color: #323333; margin: 0in;">
<span style="background-color: #cfe2f3;"><span style="font-family: inherit;">hostname: localhost</span></span></div>
<div style="color: #323333; margin: 0in;">
<span style="background-color: #cfe2f3;"><span style="font-family: inherit;">port: 1521</span></span></div>
<div style="color: #323333; margin: 0in;">
<span style="background-color: #cfe2f3;"><span style="font-family: inherit;">sid: xe</span></span></div>
<div style="color: #323333; margin: 0in;">
<span style="background-color: #cfe2f3;"><span style="font-family: inherit;">service name: xe.oracle.docker</span></span></div>
<div style="color: #323333; margin: 0in;">
<span style="background-color: #cfe2f3;"><span style="font-family: inherit;">username: system</span></span></div>
<div style="color: #323333; margin: 0in;">
<span style="background-color: #cfe2f3;"><span style="font-family: inherit;">password: oracle</span></span></div>
<div style="color: #323333; margin: 0in;">
<span style="background: #EAEEF3;"><span style="font-family: inherit;"><br /></span></span></div>
<div style="color: #555555; margin: 0in;">
<span style="font-family: inherit;"><b>To
connect using sqlplus:</b></span></div>
<div style="color: #555555; margin: 0in;">
<span style="background-color: #cfe2f3;"><span style="font-family: inherit;">sqlplus
system/oracle@//localhost:1521/xe.oracle.docker</span></span></div>
<div lang="en-US" style="margin: 0in;">
<span style="color: #555555;"><span style="font-family: inherit;"><b>To connect on mac os - install instantclient and run from there:</b></span></span></div>
<!--StartFragment-->
<!--EndFragment--><br />
<div style="color: #323333; margin: 0in;">
<span style="background-color: #cfe2f3;"><span style="font-family: inherit;">./instantclient_12_1/sqlplus system/oracle@local</span></span></div>
<span style="font-family: inherit;"><br /></span>
<br />
<span style="font-family: inherit;">To setup the initial DB I tried out <a href="https://flywaydb.org/" target="_blank">Flyway</a>. All quite simple and easy to implement. </span><br />
<span style="font-family: inherit;">Under resources/db/migration there are a number of sql files that do the initial database setup.</span><br />
<br />
<ol>
<a href="https://www.blogger.com/blogger.g?blogID=2976711854892236281" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=2976711854892236281" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=2976711854892236281" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a>
<li><span style="font-family: inherit;">create the tables: CODE and CODE_TYPE, </span></li>
<li><span style="font-family: inherit;">insert initial data </span></li>
<li><span style="font-family: inherit;">enable versioning on those tables.</span></li>
</ol>
<br />
<span style="font-family: inherit;">When you enable versioning the following happens, the table is renamed and a view is created allowing the "recording" of changes that occur.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Reference: (</span><span style="font-family: inherit;">oracle presentation available </span><a href="http://download.oracle.com/otndocs/products/workspace_manager/pdf/wrkspmgr_longtrans_s298742_120403.pdf" style="font-family: inherit;" target="_blank">here</a>)<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgHcKzJ3bIFw7NwGfsooXQnwctP7YIXrKC-Gv58nLK5-cGJH9DGNnIKO-BF77r_qlV1Mwl2dPikmgz67gX-oVhq59QTmpvobkotFxQyVrbgOzVfA4I3SkLLgF7MANmkr7-MQwB79pXBrfFG/s1600/download_oracle_com_otndocs_products_workspace_manager_pdf_wrkspmgr_longtrans_s298742_120403_pdf.png" imageanchor="1"><img border="0" height="160" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgHcKzJ3bIFw7NwGfsooXQnwctP7YIXrKC-Gv58nLK5-cGJH9DGNnIKO-BF77r_qlV1Mwl2dPikmgz67gX-oVhq59QTmpvobkotFxQyVrbgOzVfA4I3SkLLgF7MANmkr7-MQwB79pXBrfFG/s400/download_oracle_com_otndocs_products_workspace_manager_pdf_wrkspmgr_longtrans_s298742_120403_pdf.png" width="400" /></a><br />
<a href="https://www.blogger.com/blogger.g?blogID=2976711854892236281" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><span style="font-family: inherit;"></span>
<span style="font-family: inherit;"></span>
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">So after the initial setup you will seen a number of tables and views:</span><br />
<br />
<span style="font-family: inherit;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQBPqFVXrLHV1flQmRbOkrosSXWkevtTWxzS9gDQoaYNWZChJg7z9tm46yIprm7oWJ7sx77e2w3RfTMIQhDS5ZI880YeokOG5tP2YaSpP68RguGBAWibOf2GjMpPRty6xxiqBj7HAbMjf7/s1600/Oracle_Docker_sql_-_DataGrip_2016_2_2.png" imageanchor="1"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQBPqFVXrLHV1flQmRbOkrosSXWkevtTWxzS9gDQoaYNWZChJg7z9tm46yIprm7oWJ7sx77e2w3RfTMIQhDS5ZI880YeokOG5tP2YaSpP68RguGBAWibOf2GjMpPRty6xxiqBj7HAbMjf7/s200/Oracle_Docker_sql_-_DataGrip_2016_2_2.png" width="136" /></a></span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">To try this out...</span><br />
<br />
<ol>
<li>Get the Oracle Docker image</li>
<li><span style="font-family: inherit;">Once the DB is started... run the boot DataApplication.</span></li>
<li><span style="font-family: inherit;">Use Postman on the REST resources below</span></li>
</ol>
<span style="font-family: inherit;"></span><br />
<span style="font-family: inherit;"><b>To check the current workspace:</b></span><br />
<span style="background-color: #cfe2f3;">GET: http://localhost:9119/poc/workspace</span><br />
<b>To select all the information from the code table for the current workspace:</b><br />
<span style="background-color: #cfe2f3;">GET: http://localhost:9119/poc/workspace/data/code</span><br />
<b>Create code in the current workspace:</b><br />
<span style="background-color: #cfe2f3;">POST: http://localhost:9119/poc/workspace/data/code</span><br />
<span style="background-color: #cfe2f3;">BODY: </span><br />
<span style="background-color: #cfe2f3;">{</span><br />
<span style="background-color: #cfe2f3;"><span class="Apple-tab-span" style="white-space: pre;"> </span>"id":2,</span><br />
<span style="background-color: #cfe2f3;"><span class="Apple-tab-span" style="white-space: pre;"> </span>"descr": "some code",</span><br />
<span style="background-color: #cfe2f3;"><span class="Apple-tab-span" style="white-space: pre;"> </span>"type": 1</span><br />
<span style="background-color: #cfe2f3;">}</span><br />
<b>To change workspace (LIVE is default and available):</b><br />
<span style="background-color: #cfe2f3;">PUT: http://localhost:9119/poc/workspace/</span><span style="background-color: #cfe2f3;">{workspaceName}</span><br />
<b>To create a workspace:</b><br />
<span style="background-color: #cfe2f3;">POST: http://localhost:9119/poc/workspace/{workspaceName}</span><br />
<b>To remove a workspace:</b><br />
<span style="background-color: #cfe2f3;">DELETE: http://localhost:9119/poc/workspace/{workspaceName}</span><br />
<div>
<b>To merge a workspace:</b><br />
<span style="background-color: #cfe2f3;">PUT: http://localhost:9119/poc/workspace/merge/</span><span style="background-color: #cfe2f3;">{workspaceName}</span></div>
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">These REST resources just wrap some of the functions from the <a href="http://docs.oracle.com/cd/B19306_01/appdev.102/b14253/long_ref.htm#i108854" target="_blank">DBMS_WM Package</a>. </span><br />
This is maybe just the tip of a iceberg, as there is a ton of functionality available from this package.<br />
<br />
<span style="font-family: inherit;">All the code is available </span><a href="https://github.com/bdupreez/OracleWorkspaceManager" style="font-family: inherit;" target="_blank">here</a>.<br />
<br />
<br /></div>
Brianhttp://www.blogger.com/profile/09252223334818664049noreply@blogger.com29tag:blogger.com,1999:blog-2976711854892236281.post-10771498398654068062016-09-07T07:45:00.002+02:002016-09-07T07:45:55.475+02:00Re-InspiredIt has been a very long time since I last posted.<br />
Due to some crazy work pressures and a couple months of working 2 jobs, I just could not find the time nor inspiration to contribute to this blog.<br />
<br />
However today a work colleague sent me the following:
<a href="https://github.com/shekhargulati/52-technologies-in-2016">52 Technologies in 2016</a><br />
<br />
It's awesome.<br />
It reminded me how much fun I had driving myself to learn, try, demo and blog about new tech all the time.
Lets hope life does not get in the way.
Brianhttp://www.blogger.com/profile/09252223334818664049noreply@blogger.com9tag:blogger.com,1999:blog-2976711854892236281.post-13471885718219638832014-12-23T10:38:00.001+02:002014-12-23T10:38:52.864+02:00Amazon AWS Elastic Beanstalk, Python, Flask and Sci-Stack DockerThis actually took me longer than I'd like to admit to get working, but in the end the solution is quite neat and simple, so it was probably worth it, and hopefully this could save other people some time.<br />
<br />
<b>The Amazon Docker file looks like this: </b><br />
<a href="https://github.com/aws/aws-eb-python-dockerfiles/blob/master/3.4.2-aws-eb-onbuild/Dockerfile" target="_blank">AWS Elastic Beanstalk Dockerfile - Github</a><br />
<br />
This installs the contents of the root folder requirements.txt before running your Docker file. So for my application the basic "non-sci" packages could be installed simply enough.<br />
<b><br /></b>
<b>Root Folder: requirements.txt:</b>
<br />
<script class="brush: xml" type="syntaxhighlighter">
Flask
Flask-RESTful
feedparser
pymongo
twitter
requests
pygments
</script>
Then to install the sci related packages... numpy, scipy, pandas, scikit-learn and nltk. I created another requirements.txt in an aws-post-install folder. This is to be run once the Amazon linux OS has been updated and all the required OS dependencies have been installed.<br />
<br />
<b>Post Docker requirements.txt:</b>
<br />
<script class="brush: xml" type="syntaxhighlighter">
numpy
scipy
pandas
scikit-learn
nltk
</script>
My custom docker file, that builds ontop of the Amazon image looked as follows:<br />
<br />
<b>Docker File:</b><br />
<script class="brush: xml" type="syntaxhighlighter">
FROM amazon/aws-eb-python:3.4.2-onbuild-3.5.1
RUN apt-get update
RUN apt-get --yes --force-yes install gfortran libopenblas-dev liblapack-dev libatlas-dev libblas-dev
RUN apt-get --yes --force-yes install python3-dev python3-numpy python3-scipy cython3 python3-setuptools
# Add and install secondary Python modules
RUN if [ -f aws-post-install/requirements.txt ]; then /var/app/bin/pip install -r aws-post-install/requirements.txt; fi
EXPOSE 8080
#basic test
RUN /var/app/bin/python -c "import scipy, numpy, pandas, sklearn"
</script>
<br />
Next step is to get my docker image to be used directly so that the Elastic Beanstalk app doesn't have to do all the downloads and installs every time should be simple enough according the AWS you tube channel:
<a href="https://www.youtube.com/watch?v=pLw6MLqwmew">https://www.youtube.com/watch?v=pLw6MLqwmew</a>Brianhttp://www.blogger.com/profile/09252223334818664049noreply@blogger.com23tag:blogger.com,1999:blog-2976711854892236281.post-70500009784448858422014-08-26T06:13:00.002+02:002014-08-26T06:15:23.597+02:00Why Jython when you can microservice with FlaskOver the last little while I have been working on <a href="http://sibbly.com/">Sibbly</a> it's my little pet project to try summarize, group, filter and target software development information on the web. All 'n all a rather ambitious task, but the worst thing that could happen is that I learn something, so there is really no risk.
It is still currently in a very closed beta, only occasionally showing it to fellow work colleagues and getting some input.
<br />
<br />
After initially starting development for <a href="http://sibbly.com/">Sibbly</a> on Ubuntu, as I was always planning on deploying on Ubuntu, I had migrated back to windows, and after a couple weeks of work when finally deploying to Ubuntu... Surprise! it obviously did work right off the bat.
<br />
<br />
The issue I ended up with was, there seems to be a classpath issue between <a href="http://projects.spring.io/spring-boot/">Spring Boot</a>, it's embedded Tomcat instance and Jython. The reason I use Jython is for an awesome library called <a href="http://pygments.org/">Pygments</a>.<br />
<br />
So after much dismay and checking all the Java alternatives and attempted Pygment ports (jygments, jgments), I started thinking of alternate solutions.<br />
Having recently read: <a href="http://martinfowler.com/articles/microservices.html">Microservices</a> I decided to look at a way of interacting with Python more indirectly.
<br />
This lead me to: <a href="http://flask.pocoo.org/" target="_blank">Flask</a><br />
Within a couple minutes thanks to:
<a href="http://blog.miguelgrinberg.com/post/designing-a-restful-api-with-python-and-flask">Awesome Flask Example</a><br />
I had the following up and running:<br />
<br />
<script src="https://gist.github.com/bdupreez/4920aae3be84aa7e6716.js"></script>
<br />
What this little bit of Python does is wrap and expose the highlight and guess functionality from Pygments via a RESTful service accepting and producing JSON.<br />
<br />
I deploy Sibbly on <a href="https://www.digitalocean.com/">DigitalOcean</a><br />
To install Python on my droplet, I followed the process below:
<br />
<pre>
sudo apt-get install python-dev build-essential
sudo apt-get install zlib1g-dev
sudo apt-get install libssl-dev openssl
sudo apt-get install python-pip
sudo pip install virtualenv
sudo pip install virtualenvwrapper
export WORKON_HOME="$HOME/.virtualenvs"
source /usr/local/bin/virtualenvwrapper.sh
sudo mkdir /opt/python3.4.1
wget http://python.org/ftp/python/3.4.1/Python-3.4.1.tgz
tar xvfz Python-3.4.1.tgz
cd Python-3.4.1
./configure --prefix=/opt/python3.4.1
make
sudo make install
mkvirtualenv --python /opt/python3.4.1/bin/python3 py-3.4.1
workon py-3.4.1
pip install flask
pip install pygments
</pre>
Once that was done to run the Flask app:<br />
<pre>
python app.py & disown
</pre>Brianhttp://www.blogger.com/profile/09252223334818664049noreply@blogger.com16tag:blogger.com,1999:blog-2976711854892236281.post-33527998084575140122014-08-10T19:09:00.000+02:002014-08-10T19:15:59.079+02:00Upgrading Spring 3.x and Hibernate 3.x to Spring Platform 1.0.1 (Spring + hibernate 4.x)I recent volunteered to upgrade our newest project to the latest version of Spring Platform. What Spring Platform gives you is dependency & plugin management across the whole Spring framework's set of libraries.<br />
<div>
<br /></div>
<div>
Since we had fallen behind a little the upgrade did raise some funnies. Here are the things I ran into:<br />
<br />
<b><u>Maven:</u></b><br />
Our pom files were still referencing:<br />
<i>hibernate.jar </i><br />
<i>ehcache.jar </i><br />
These artefacts don't exit on the latest version, so replaced those with<br />
<i>hibernate-core.jar </i>and <i>ehcache-core.jar</i><br />
<br />
We also still use the hibernate tools + maven run plugin to reverse engineer our db object.<br />
This I needed to update to a release candidate:<br />
<br />
<div>
<script class="brush: xml" type="syntaxhighlighter">
<hibernate-tools .version="">4.3.1.CR1</hibernate-tools>
</script>
</div>
<div>
<br /></div>
<div>
<b><u>Hibernate:</u></b></div>
<div>
</div>
The code: "Hibernate.createBlob"... no longer exists<br />
<br />
replaced with:<br />
<script class="brush: java" type="syntaxhighlighter">
private Blob createBlob(final byte[] bytes) {
return NonContextualLobCreator.INSTANCE.wrap(NonContextualLobCreator.INSTANCE.createBlob(bytes));
}
</script>
<div>
<br />
On the HibernateTemplate<br />
return types are now List; not element...So needed to add casts for the lists being returned.<br />
<div>
<br />
<div>
<div>
import org.hibernate.classic.Session;<br />
replaced with:<br />
import org.hibernate.Session;<br />
<br />
Reverse engineer works a little differently...</div>
Assigns Long to numeric...<br />
Added:<br />
<script class="brush: xml" type="syntaxhighlighter">
<type-mapping>
<sql-type jdbc-type="NUMERIC" precision="4" hibernate-type="java.lang.Integer" />
<sql-type jdbc-type="NUMERIC" precision="6" hibernate-type="java.lang.Integer" />
<sql-type jdbc-type="NUMERIC" precision="8" hibernate-type="java.lang.Integer" />
<sql-type jdbc-type="NUMERIC" precision="10" hibernate-type="java.lang.Long" />
<sql-type jdbc-type="DECIMAL" precision='4' scale='0' hibernate-type="java.lang.Integer" not-null="true"/>
<sql-type jdbc-type="DECIMAL" precision='6' scale='0' hibernate-type="java.lang.Integer" not-null="true"/>
<sql-type jdbc-type="DATE" hibernate-type="java.util.Date"/>
</type-mapping>
</script>
<br />
<div>
<div>
<b>Possible Errors:</b><br />
<br />
<ul>
<li>Caused by: org.hibernate.service.UnknownUnwrapTypeException: Cannot unwrap to requested type [javax.sql.DataSource]</li>
</ul>
Add a dependency for c3p0:
<br />
<script class="brush: xml" type="syntaxhighlighter">
<dependency>
<groupid>org.hibernate</groupId>
<artifactid>hibernate-c3p0</artifactId>
<version>${hibernate.version}</version>
</dependency>
</script>
</div>
<br />
And configure the settings in the cfg.xml for it:
<script class="brush: xml" type="syntaxhighlighter">
<property name="hibernate.c3p0.min_size">5</property>
<property name="hibernate.c3p0.max_size">20</property>
<property name="hibernate.c3p0.timeout">300</property>
<property name="hibernate.c3p0.max_statements">50</property>
<property name="hibernate.c3p0.idle_test_period">3000</property>
</script>
<br />
<ul>
<li>Caused by: java.lang.ClassNotFoundException: org.hibernate.engine.FilterDefinition</li>
</ul>
<br />
<div>
Probably still using a reference to hibernate3 factory / bean somewhere, change to hibernate4:<br />
org.springframework.orm.hibernate3.LocalSessionFactoryBean<br />
org.springframework.orm.hibernate3.HibernateTransactionManager<br />
<br /></div>
<ul>
<li>Caused by: java.lang.ClassNotFoundException: Could not load requested class : org.hibernate.hql.classic.ClassicQueryTranslatorFactory
There is minor change in new APIs, so this can be resolved by replacing property value with:</li>
</ul>
<br />
org.hibernate.hql.<b>internal</b>.classic.ClassicQueryTranslatorFactory.
<br />
<br />
<b><u>Spring:</u></b><br />
Amazingly some of our application context files still referenced the Spring DTD ... replaced with XSD<br />
<div>
<script class="brush: xml" type="syntaxhighlighter">
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://www.springframework.org/schema/beans"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
</script>
<br />
<div>
<div>
In Spring configs added for c3p0:<br />
<script class="brush: xml" type="syntaxhighlighter">
<prop key="hibernate.c3p0.min_size">5</prop>
<prop key="hibernate.c3p0.max_size">20</prop>
<prop key="hibernate.c3p0.timeout">300</prop>
<prop key="hibernate.c3p0.max_statements">50</prop>
<prop key="hibernate.c3p0.idle_test_period">3000</prop>
</script>
<div>
<br clear="none" />
Spring removed the "local"=: so needed to just change that to "ref"= <br />
<br />
<div>
<div>
<div>
Spring HibernateDaoSupport no longer has: "releaseSession(session);", which is a good thing so was forced to update the code to work within a transaction. <br />
<div>
<br /></div>
</div>
</div>
<div>
<div>
<div>
<b>Possible Errors:</b></div>
<div>
<br />
<ul>
<li>getFlushMode is not valid without active transaction; nested exception is org.hibernate.HibernateException: getFlushMode is not valid without active transaction</li>
</ul>
</div>
<div>
<br clear="none" /></div>
<div>
Removed from hibernate properties:</div>
<div>
<prop key="hibernate.current_session_context_class">thread</prop></div>
<div>
<br clear="none" /></div>
<div>
<div>
<table border="1" summary="Miscellaneous Properties">
<tbody>
<tr>
<td colspan="1" rowspan="1">Supply a custom strategy for the scoping of the "current"<code>Session</code>. See <a href="http://docs.jboss.org/hibernate/core/3.3/reference/en/html/architecture.html%23architecture-current-session" shape="rect" title="2.5. Contextual sessions">Section 2.5, “Contextual sessions”</a> for more information about the built-in strategies</td>
</tr>
</tbody>
</table>
</div>
</div>
<div>
<br />
<ul>
<li>org.springframework.dao.InvalidDataAccessApiUsageException: Write operations are not allowed in read-only mode (FlushMode.MANUAL): Turn your Session into FlushMode.COMMIT/AUTO or remove 'readOnly' marker from transaction definition.</li>
</ul>
</div>
<div>
<br /></div>
<div>
Another option is :</div>
<div>
<bean id ="productHibernateTemplate" class="org.springframework.orm.hibernate4.HibernateTemplate"><br />
<property name="sessionFactory" ref="productSessionFactory"/><br />
<strong><property name="checkWriteOperations" value="false"/></strong><br />
</bean></div>
<div>
<br clear="none" /></div>
<div>
<ul>
<li>java.lang.NoClassDefFoundError: javax/servlet/SessionCookieConfig</li>
</ul>
</div>
<div>
Servlet version update:</div>
<script class="brush: xml" type="syntaxhighlighter">
<dependency>
<groupid>javax.servlet</groupId>
<artifactid>servlet-api</artifactId>
<version>3.0.1</version>
</dependency>
</script>
<div>
<br />
<ul>
<li>Then deploying on weblogic javassist: $$_javassist_ cannot be cast to javassist.util.proxy.Proxy</li>
</ul>
</div>
<div>
<br clear="none" /></div>
<div>
The issue here was that there were different versions of javassist being brought into the ear. I all references removed from all our poms, so that the correct version gets pulled in from from Spring/Hibernate...</div>
<div>
<br clear="none" /></div>
<div>
and then configured weblogic to prefer our version:</div>
<div>
<br clear="none" /></div>
<script class="brush: xml" type="syntaxhighlighter">
<?xml version="1.0" encoding="UTF-8"?>
<weblogic-application>
<application-param>
<param-name>webapp.encoding.default</param-name>
<param-value>UTF-8</param-value>
</application-param>
<prefer-application-packages>
<package-name>javax.jws.*</package-name>
<package-name>org.apache.xerces.*</package-name>
<package-name>org.apache.xalan.*</package-name>
<package-name>org.apache.commons.net.*</package-name>
<package-name>org.joda.*</package-name>
<package-name>javassist.*</package-name>
</prefer-application-packages>
</weblogic-application>
</script>
<br />
<div>
<br clear="none" /></div>
<div>
<br clear="none" /></div>
<div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
Brianhttp://www.blogger.com/profile/09252223334818664049noreply@blogger.com13tag:blogger.com,1999:blog-2976711854892236281.post-2642520866165488012014-07-19T10:02:00.000+02:002014-07-19T10:10:43.309+02:00TDD, Hamcrest, ShazamcrestRecently we have started to try get a more TDD culture started at work, having always believed in thorough testing and decent code coverage it shouldn't have been too hard. However... teaching a old dog new tricks can sometimes require quite a bit of patience. Turns out breaking coding habits formulated of more than a decade of keyboard bashing is harder than it seems.<br />
<br />
So with generating an enormous amount of test code, comes the usual task code & test maintenance and reuse.<br />
One of the tools / libraries we have included is <a href="http://hamcrest.org/JavaHamcrest/" target="_blank">Hamcrest</a>, which not only improves the readability of assertion failures, but allows you to create and extend custom matchers, which you can then reuse across multiple test scenarios.<br />
<br />
I am not going to go into too much detail on Hamcrest here, where are a bunch of great resources / blogs / tutorials out there.. just a few:<br />
<a href="http://www.baeldung.com/hamcrest-collections-arrays" target="_blank">http://www.baeldung.com/hamcrest-collections-arrays</a><br />
<a href="https://weblogs.java.net/blog/johnsmart/archive/2011/12/12/some-useful-new-hamcrest-matchers-collections" target="_blank">https://weblogs.java.net/blog/johnsmart/archive/2011/12/12/some-useful-new-hamcrest-matchers-collections</a><br />
<a href="http://edgibbs.com/junit-4-with-hamcrest/" target="_blank">http://edgibbs.com/junit-4-with-hamcrest/</a><br />
<a href="http://www.planetgeek.ch/2012/03/07/create-your-own-matcher/" target="_blank">http://www.planetgeek.ch/2012/03/07/create-your-own-matcher/</a><br />
<br />
While creating a custom type safe matcher for one of our domain objects, I realised that was insane.. really.. this.getA == that.getA... mmmm no.<br />
So I went searching for something could help and and after a bit, I found: <a href="https://github.com/shazam/shazamcrest/" target="_blank">Shazamcrest</a> (bonus points for the name)<br />
What <a href="https://github.com/shazam/shazamcrest/" target="_blank">Shazamcrest</a> does is:<br />
Serialize the objects to compare.<br />
Compares them and then on fail throws a ComparisonFailure, which the major IDE's allow you use their build in diff display.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgG-XHc2gWRvD-hFjSAYAxIrvIDV5wUZdRmp1ywruvJ3f0xRKbJfiBkmGCu-q9wQm5R4BEd3S_u7jFG_bWvmpWbMUHXFmffl4t_WuzJvTLJ2urQn7vnSE5sXtOcm64ufJ0GzLptWSTE9qaA/s1600/intellij+compare.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgG-XHc2gWRvD-hFjSAYAxIrvIDV5wUZdRmp1ywruvJ3f0xRKbJfiBkmGCu-q9wQm5R4BEd3S_u7jFG_bWvmpWbMUHXFmffl4t_WuzJvTLJ2urQn7vnSE5sXtOcm64ufJ0GzLptWSTE9qaA/s1600/intellij+compare.png" height="253" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
Great... no manual bean compares.</div>
<div class="separator" style="clear: both; text-align: left;">
So I add the maven dependency, try it out on our complex domain object....</div>
<div class="separator" style="clear: both; text-align: left;">
StackOverflowError.... It was a known limitation at the time. The json provider <a href="https://github.com/shazam/shazamcrest/" target="_blank">Shazamcrest</a> was using: </div>
<div class="separator" style="clear: both; text-align: left;">
<a href="https://code.google.com/p/google-gson/" target="_blank">GSON</a> does not cater for circular reference serialization.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
As both <a href="https://github.com/shazam/shazamcrest/" target="_blank">Shazamcrest</a> and <a href="https://code.google.com/p/google-gson/" target="_blank">GSON</a> being opensource, I decided to have a look and see if I could contribute, anything is better that writing a manual bean matcher. After some investigation I found that the guys on the <a href="https://code.google.com/p/google-gson/" target="_blank">GSON</a> project have created a fix <a href="https://code.google.com/p/google-gson/source/browse/trunk/extras/src/main/java/com/google/gson/graph/GraphAdapterBuilder.java" target="_blank">GraphAdapterBuilder</a>, it is just not distributed with the actual library.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
So after fork on the <a href="https://github.com/shazam/shazamcrest/" target="_blank">Shazamcrest</a> GitHub project, a little bit of code and submitting a pull request:</div>
<div class="separator" style="clear: both; text-align: left;">
<a href="https://github.com/shazam/shazamcrest/pull/5">https://github.com/shazam/shazamcrest/pull/5</a></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
The guys on the <a href="https://github.com/shazam/shazamcrest/" target="_blank">Shazamcrest</a> project very quickly merged my changes in and published a new version to the maven repo (Thanks for that). </div>
<div class="separator" style="clear: both; text-align: left;">
So be sure to use the 0.8 version if you are struggling with circular references.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<br />Brianhttp://www.blogger.com/profile/09252223334818664049noreply@blogger.com8tag:blogger.com,1999:blog-2976711854892236281.post-36763936765987743942014-05-26T21:49:00.001+02:002014-05-26T21:49:50.758+02:00Playing with Java 8 - Lambdas, Paths and FilesI needed to read a whole bunch of files recently and instead of just grabbing my old FileUtils.java that I and probably most developers have and then copy from project to project, I decided to have quick look at how else to do it...<br />
Yes, I know there is <a href="http://commons.apache.org/proper/commons-io/" target="_blank">Commons IO</a> and <a href="https://code.google.com/p/guava-libraries/" target="_blank">Google IO</a>, why would I even bother? They probably do it better, but I wanted to check out the NIO jdk classes and play with lambdas aswell.. and to be honest, I think this actually ended up being a very neat bit of code.<br />
<br />
So I had a specific use case:<br />
I wanted to read all the source files from a whole directory tree, line by line.<br />
<br />
What this code does, it uses Files.walk to recursively get all the paths from the starting point, it creates a stream, which I then filter to only files that end with the required extension. For each of those files, I use Files.lines to create a stream of Strings, one per line. I trim that, filter out the empty ones and add them to the return collection.<br />
All very concise thanks to the new constructs.<br />
<br />
<br />
<script src="https://gist.github.com/bdupreez/bd422f214f2058bf2c7a.js"></script>
<br />Brianhttp://www.blogger.com/profile/09252223334818664049noreply@blogger.com9tag:blogger.com,1999:blog-2976711854892236281.post-59395962237804091542014-04-26T14:56:00.000+02:002014-04-26T14:56:59.515+02:00Playing with Java 8 - Lambdas and ConcurrencySo Java 8 was released a while back, with a ton of features and changes. All us Java zealots have been waiting for this for ages, all the way back to from when they originally announced all the great features that will be in Java 7, which ended up being pulled.<br />
<br />
I have just recently had the time to actually start giving it a real look, I updated my home projects to 8 and I have to say I am generally quite happy with what we got. The java.time API the "mimics" JodaTime is a big improvement, the java.util.stream package is going useful, lambdas are going to change our coding style, which might take a bit of getting used to and with those changes... the quote, "With great power comes great responsibility" rings true, I sense there may be some interesting times in our future, as is quite easy to write some hard to decipher code. As an example debugging the code I wrote below would be "fun"...<br />
<br />
The file example is on my <a href="https://github.com/bdupreez/Blog" target="_blank">Github blog repo</a><br />
<br />
What this example does is simple, run couple threads, do some work concurrently, then wait for them all to complete. I figured while I am playing with Java 8, let me go for it fully...<br />
Here's what I came up with:<br />
<script src="https://gist.github.com/bdupreez/4652ed0b6324d1f70036.js"></script>
Test:<br />
<script src="https://gist.github.com/bdupreez/cc4cbf0c6a0caa2bf96a.js"></script>
Output:<br />
<br />
0 [pool-1-thread-1] Starting: StringInputTask{taskName='Task 1'}<br />
0 [pool-1-thread-5] Starting: StringInputTask{taskName='Task 5'}<br />
0 [pool-1-thread-2] Starting: StringInputTask{taskName='Task 2'}<br />
2 [pool-1-thread-4] Starting: StringInputTask{taskName='Task 4'}<br />
2 [pool-1-thread-3] Starting: StringInputTask{taskName='Task 3'}<br />
3003 [pool-1-thread-5] Done: Task 5<br />
3004 [pool-1-thread-3] Done: Task 3<br />
3003 [pool-1-thread-1] Done: Task 1<br />
3003 [pool-1-thread-4] Done: Task 4<br />
3003 [pool-1-thread-2] Done: Task 2<br />
3007 [Thread-0] WaitingFuturesRunner - complete... adding results<br />
<br />
<br />
Some of the useful articles / links I found and read while doing this:<br />
<br />
Oracle: <a href="http://www.oracle.com/webfolder/technetwork/tutorials/obe/java/Lambda-QuickStart/index.html#overview" target="_blank">Lambda Tutorial</a><br />
IBM: <a href="http://www.ibm.com/developerworks/library/j-jvmc2/index.html" target="_blank">Java 8 Concurrency</a><br />
Tomasz Nurkiewicz : <a href="http://www.nurkiewicz.com/2013/05/java-8-definitive-guide-to.html" target="_blank">Definitive Guide to CompletableFuture</a><br />
<br />
<br />Brianhttp://www.blogger.com/profile/09252223334818664049noreply@blogger.com7tag:blogger.com,1999:blog-2976711854892236281.post-42305888637191597422014-02-16T15:07:00.000+02:002014-02-16T15:08:02.722+02:00Local Wikipedia with Solr and Spring Data Continuing with my little AI / Machine Learning research project... I wanted to have a decent sized repo of English text, that was not in a complete mess like a large percentage of data on the internet. I figured I would try Wikipedia, but what to do with about 40Gb of XML? how do I work / query with all that data. I figured based on recent work implementation where we load something like 200 000 000 records on into a Solr cache, Solr would be the way to go, so the is an example of my basic implementation.<br />
<br />
Required for this example:<br />
<br />
<a href="http://download.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2">Wikipedia download</a> (warning it is a 9.9Gb file, extracts to about 42Gb)<br />
<a href="https://lucene.apache.org/solr/">Solr</a><br />
<a href="http://projects.spring.io/spring-data/">Spring Data</a> (Great Blog / Examples on Spring Data: <a href="http://www.petrikainulainen.net/spring-data-solr-tutorial/">Petri Kainulainen's blog</a>)<br />
<br />
All the code and unit test for this post is on my blog <a href="https://github.com/bdupreez/Blog">GitHub Repo</a><br />
<br />
When setting up Solr from scratch, you can have a look at Solr's <a href="http://wiki.apache.org/solr/">wiki</a> or documentation, their documentation is pretty good. There is also an example of importing Wikipedia <a href="http://wiki.apache.org/solr/DataImportHandler">here</a>, I started with that and made some minor modifications.<br />
<br />
For this specific example the Solr config needed (<core>/conf):</core><br />
For this example (and in the below config files), <br />
Solr home: /Development/Solr<br />
Index / Data: /Development/Data/solr_data/wikipedia<br />
Import File: /Development/Data/enwiki-latest-pages-articles.xml<br />
<br />
The full import into Solr took about 48 hours on my old 2011 i5 iMac and the index on my current setup is about 52Gb.<br />
<br />
Data Config for the import:<br />
<br />
<script src="https://gist.github.com/bdupreez/80a6f2feef0b0676acdc.js"></script>
Schema:<br />
<br />
<script src="https://gist.github.com/bdupreez/45e371ab225d354d885b.js"></script>
Solr Config:<br />
<br />
<script src="https://gist.github.com/bdupreez/93cc2405f206eeda8cf8.js"></script>
The code for this ended up being quite clean, Spring Data - Solr, gives 2 main interfaces SolrIndexService, and SolrCrudRespository, you simply extend / implement these 2, wrap that in a single interface, autowire from a Spring Java context and you good to go.<br />
<br />
Repository:<br />
<br />
<script src="https://gist.github.com/bdupreez/dae5efbcc9f66307ed93.js"></script>
IndexService:<br />
<br />
<script src="https://gist.github.com/bdupreez/5347166b3087f03b562d.js"></script>
SolrService:<br />
<br />
<br />
<script src="https://gist.github.com/bdupreez/da7a26f50946d34eb914.js"></script>
SpringContext:<br />
<br />
<script src="https://gist.github.com/bdupreez/a758358cd49a2963f316.js"></script>
Next thing for me to look at for sourcing data is Spring Social.Brianhttp://www.blogger.com/profile/09252223334818664049noreply@blogger.com5tag:blogger.com,1999:blog-2976711854892236281.post-30961345761928039062014-01-12T08:57:00.003+02:002014-01-12T08:57:51.099+02:00BYG (Bing, Yahoo, Google) Search WrapperOne small section of my Aria project will be to interface with the current search engines out there. To do this I will require a module that will give me a consistent interface to work with the 3 main providers; Bing, Yahoo! and Google. (and any future ones I may want to add). This is a basic example or that module.<br />
<br />
First thing required is to set up accounts / projects and the like with the relevant providers.<br />
I won't describe this process as they were all pretty well documented.<br />
<br />
<a href="http://www.bing.com/dev/en-us/dev-center">Bing Developer Center</a><br />
<a href="http://developer.yahoo.com/">Yahoo Developer Network</a><br />
<a href="https://cloud.google.com/console">Google Developers Console</a><br />
<br />
<br />
A couple tips for the above sites.<br />
<br />
<ul>
<li>Bing: Setup both the web and synonym searches.</li>
<li>Yahoo: In the BOSS console, under manage account, put in a daily limit $ amount (or turn of limit), as they only allow 1 free query a day... so only the first request works.</li>
<li>Google: It doesn't seem that you can set it up to search the whole web, but after creating your custom search engine, you can select "Search the entire web but emphasize included sites" so don't worry about that.</li>
</ul>
<div>
<br /></div>
<div>
All these providers allow for many options while searching ( e.g. images, location, news, video etc.) , however in this initial example I have limited it to just a pure and simple web search.</div>
<div>
<br /></div>
<div>
All the code will be available in my <a href="https://github.com/bdupreez/Blog">blog Github repository</a>.</div>
<div>
<br /></div>
<div>
Going through the main points.</div>
<div>
There is a BasicWebSearch interface, that takes the search term and returns SearchResults. </div>
<div>
SearchResults contains results in a map based on a result type enum. </div>
<div>
The implementations of BasicWebSearch namely: BingSearch, GoogleSearch and YahooSearch call the relevant search engine with the search term and then convert the results into a SearchResult. In the case of Yahoo and Bing, I map the JSON result to the SearchResult. Google however does that in their search client included in the dependencies.</div>
<div>
<br /></div>
<div>
Now for the main code bits:</div>
<div>
<br /></div>
<div>
<b>SearchSettings</b></div>
<div>
As this is just an example, I use included the search settings in the following class, be sure to replace with the relevant values.</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/bdupreez/dfa95300a426b5154fb6.js"></script></div>
<div>
<b>UrlConnectionHandler</b></div>
<div>
As both Bing and Yahoo use an HttpUrlConnection, I figured I would centralise the handling of that, the only difference between the 2 is that Bing used basic authentication and Yahoo I went with the OAuth implementation.</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/bdupreez/a424b536076cecbf5b5d.js"></script></div>
<div>
<b>BingSearch</b></div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/bdupreez/c7f4aee7a01cb9a5443a.js"></script></div>
<div>
<b>BingResultParser</b></div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/bdupreez/2beeb902679ee9c7ff8f.js"></script></div>
<div>
<b>YahooSearch</b></div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/bdupreez/66da862f01cfb796d4d7.js"></script></div>
<div>
<b>YahooResultParser</b></div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/bdupreez/f6ce41d7fca8e4d1d4b0.js"></script></div>
<div>
<b>GoogleSearch</b></div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/bdupreez/7475022fa1e18a32f527.js"></script></div>
<div>
<b>GoogleSearchResult</b><br />
Google has a whole bunch of extra information being returned so I extended the base SearchResult so add all the information just in case I ever need it.<br />
<br />
<script src="https://gist.github.com/bdupreez/acd9940633a46e489cda.js"></script>
<b>Maven Dependencies</b></div>
<div>
<br /></div>
<script src="https://gist.github.com/bdupreez/7661b24144cbb564d2e4.js"></script>
<div>
<br /></div>
Brianhttp://www.blogger.com/profile/09252223334818664049noreply@blogger.com4tag:blogger.com,1999:blog-2976711854892236281.post-416652063229776582013-12-15T08:40:00.000+02:002013-12-15T08:48:02.627+02:00Predicting the next most probable part of speech.I have recently been spending some of my spare time learning and about AI and machine learning , after a couple books, a bunch of tutorials and most of <a href="https://www.coursera.org/" target="_blank">Andrew Ng's Coursera</a> course. I decided enough with the theory, time for some real code.<br />
During all my late night reading I also stumbled across some of the following. <a href="http://www.loebner.net/Prizef/loebner-prize.html" target="_blank">The Loebner Prize</a>, and it's most recent winner <a href="http://www.mitsuku.com/" target="_blank">Mitsuku</a>, <a href="http://alice.pandorabots.com/" target="_blank">A.L.I.C.E </a>and <a href="http://www.cleverbot.com/" target="_blank">Cleverbot</a> and to be honest, maybe given my naivety of the field of AI, I expected much more from the above "AI" / technology, most of these current chatbots are easily confused and honestly not very impressive.<br />
Thankfully I also found Eric Horvitz's <a href="http://blogs.technet.com/b/next/archive/2013/11/15/meet-microsoft-researcher-eric-horvitz-s-ai-personal-assistant.aspx#.UpHn2K8NOTk.twitter" target="_blank">video </a>of his AI personal assistant, which resonated with what I wanted to achieve with my ventures into AI. <br />
<br />
So, with human interaction as a focus point, I started designing: "Aria" - (<i>Artificially Intelligent Research Assistant - in reverse</i>). Which, since most of my development experience is based in the Java Enterprise environment, will be built on a distributed enterprise scale using the amazing technologies that it offers to mention some Hadoop, Spark, Mahout, Solr, MySQL, Neo4J, Spring...<br />
<br />
My moonshot/daydream goal is to better the interactions of people and computers, but in reality if I only learn to use, implement and enjoy all that is involved with AI and ML, I will see myself as successful.<br />
<br />
So, for my first bit of functional machine learning...<br />
<br />
Predicting the next most probable part of speech. One of the issues with natural language processing is that words used in different contexts end up having different meanings and synonyms. To try assist with this I figured I would train a neural network with the relevant parts of speech, and then use that to assist in understanding user submitted text.<br />
<br />
This full code for this example is available on <a href="https://github.com/bdupreez/Blog/tree/master/General-ML-AI" target="_blank">Github</a>.<br />
<br />
I used a number of Java open source libraries for this:<br />
<a href="http://www.heatonresearch.com/encog" target="_blank">Encog</a><br />
<a href="http://neuroph.sourceforge.net/" target="_blank">Neuroph </a><br />
<a href="http://nlp.stanford.edu/index.shtml" target="_blank">Stanford NLP</a><br />
<a href="https://code.google.com/p/guava-libraries/" target="_blank">Google Guava</a><br />
<br />
I used a dataset of 29 000 English sentences that I sourced from a bunch of websites and open corpus's. I won't be sharing those as I have no clue what the state of the copyright is, so unfortunately to recreate this you'd need to source our own data.<br />
<br />
For the neural network implementation, I tried both Neuroph and Encog. Neuroph got my attention first with their great UI to allow me to experiment with my neural network visually in the beginning, but as soon as I created my training data with ended up being about 300MB of 0's and 1's it fell over and didn't allow me to use it. I then began looking at Encog again as I had used initially when just starting to read about ML and AI<br />
<br />
When using Neuroph in code it worked with the dataset, but then only with BackPropagation the ResilientPropagation implementation never seemed to return.<br />
So I ended up much preferring <a href="http://www.heatonresearch.com/encog" target="_blank">Encog</a>, it's resilient propagation implementation (<a href="http://www.heatonresearch.com/wiki/Resilient_Propagation" target="_blank">iRPROP+</a>) worked well and reduces the network error to about 0.018 in under 100 iterations, without me having to fine tune the settings and network architecture.<br />
<br />
How this works, I take text data, I use the Stanford NLP library to generate a list of the parts of speech in the document. I translate their Annotation into an internal enum, and then use that to build up a training data set. I persist that to file currently, just to save some time while testing. I then train and persist the neural network and test it.<br />
<br />
The Parts of Speech Enum:<br />
<br />
<script src="https://gist.github.com/bdupreez/9b39258a8e75ce7aed8a.js"></script>
The creation of the training data:<br />
<br />
<script src="https://gist.github.com/bdupreez/43312788607ec6850cdf.js"></script>
Train the network:<br />
<br />
<script src="https://gist.github.com/bdupreez/d6bd42b0cc9f17be24af.js"></script>
Test:<br />
<br />
<script src="https://gist.github.com/bdupreez/30cf1e4ae3bfd23e882f.js"></script>
Brianhttp://www.blogger.com/profile/09252223334818664049noreply@blogger.com5tag:blogger.com,1999:blog-2976711854892236281.post-32142451418581827662013-10-13T20:13:00.000+02:002013-12-15T08:41:04.504+02:00Setting up multiple versions of Python on Ubuntu I recently switched from using a Mac back to a PC, I had originally planned to use both windows and linux via dual-boot, but having purchased a Radeon and Ubuntu not even starting from the bootable USB, I decided to try run my Python development environment on Windows. After playing with python on Windows, I found it quite tedious to have both a 2.7.5 and a 3.3.2 environment. I also didn't like having to rely on <a href="http://www.lfd.uci.edu/~gohlke/pythonlibs/" target="_blank">http://www.lfd.uci.edu/~gohlke/pythonlibs/</a> for all the 'pain' free install, since trying to compile some the libs with the required C++ compiler is even a bigger pain.<br />
<br />
So I went with a colleagues suggestion of VMWare Player 6, and installed Ubuntu.<br />
<br />
After breaking a couple installs and recreating VMs left and right, I finally have a process to install and work with multiple versions of Python.<br />
<br />
<u>First up, get a whole bunch of dependencies:</u><br />
<i><b>sudo apt-get install python-dev build-essential </b></i><br />
<i><b>sudo apt-get install python-pip</b></i><br />
<i><b>sudo apt-get install libsqlite3-dev sqlite3</b></i><br />
<i><b>sudo apt-get install libreadline-dev libncurses5-dev </b></i><br />
<i><b>sudo apt-get install libssl1.0.0 tk8.5-dev zlib1g-dev liblzma-dev</b></i><br />
<i><b>sudo apt-get build-dep python2.7</b></i><br />
<i><b>sudo apt-get build-dep python3.3</b></i><br />
<i><b><br /></b></i>
<i><b>sudo pip install virtualenv</b></i><br />
<i><b>sudo pip install virtualenvwrapper</b></i><br />
<i><b><br /></b></i>
<u>Add the virtualenvwrapper settings to ~.bashrc:</u><br />
<b><i>export WORKON_HOME="$HOME/.virtualenvs"</i></b><br />
<b><i>source /usr/local/bin/virtualenvwrapper.sh</i></b><br />
<i><b><br /></b></i>
<u>Then for Python 2.7:</u><br />
<b><i>sudo mkdir /opt/python2.7.5</i></b><br />
<b><i><br /></i></b>
<b><i>wget http://python.org/ftp/python/2.7.5/Python-2.7.5.tgz</i></b><br />
<b><i>tar xvfz Python-2.7.5.tgz</i></b><br />
<b><i>cd Python-2.7.5/</i></b><br />
<b><i>./configure --prefix=/opt/python2.7.5</i></b><br />
<b><i>make</i></b><br />
<b><i>sudo make install</i></b><br />
<b><i><br /></i></b>
<b><i>mkvirtualenv --python /opt/python2.7.5/bin/python2 v-2.7.5</i></b><br />
<br />
<u>Then for Python 3.3:</u><br />
<b><i>sudo mkdir /opt/python3.3.2</i></b><br />
<b><i><br /></i></b>
<b><i>wget http://python.org/ftp/python/3.3.2/Python-3.3.2.tgz</i></b><br />
<b><i>tar xvfz Python-3.3.2.tgz</i></b><br />
<b><i>cd Python-3.3.2</i></b><br />
<b><i>./configure --prefix=/opt/python3.3.2</i></b><br />
<b><i>make </i></b><br />
<b><i>sudo make install</i></b><br />
<div>
<b><i><br /></i></b></div>
<div>
<b><i>mkvirtualenv --python /opt/python3.3.2/bin/python3 v-3.3.2</i></b></div>
<div>
<br /></div>
<div>
<u>To view the virtual environments:</u></div>
<b><i>lsvirtualenv</i></b><br />
<b><i><br /></i></b>
<u>To change between them:</u><br />
<i><b>workon [env name] </b>e.g. v-3.3.2</i><br />
<br />
<u>Then to install some of the major scientific and machine learning related packages:</u><br />
<b><i>pip install numpy</i></b><br />
<b><i>pip install ipython[all]</i></b><br />
<b><i>pip install cython</i></b><br />
<b><i>sudo apt-get build-dep python-scipy</i></b><br />
<b><i>pip install scipy</i></b><br />
<b><i>pip install matplotlib</i></b><br />
<b><i>pip install scikit-learn</i></b><br />
<b><i>pip install pandas</i></b><br />
<br />
To stop working on a particular version:<br />
<b><i>deactivate</i></b>Brianhttp://www.blogger.com/profile/09252223334818664049noreply@blogger.com5tag:blogger.com,1999:blog-2976711854892236281.post-77224991983018256622013-09-15T09:34:00.000+02:002013-09-15T09:34:15.950+02:00Wordle... so nicely doneDiscovered <a href="http://www.wordle.net/" target="_blank">Wordle</a> this morning, pointed to my blog... guess my recent posts really haven't been about java much :)<br />
<br />
<a href="http://www.wordle.net/show/wrdl/7053766/My_blog_-_recent_posts"
title="Wordle: My blog - recent posts"><img
src="http://www.wordle.net/thumb/wrdl/7053766/My_blog_-_recent_posts"
alt="Wordle: My blog - recent posts"
style="padding:4px;border:1px solid #ddd"></a>
<br />Brianhttp://www.blogger.com/profile/09252223334818664049noreply@blogger.com4tag:blogger.com,1999:blog-2976711854892236281.post-77935288917317203052013-09-07T23:58:00.001+02:002013-09-08T09:45:22.068+02:00Sourcing Twitter data, based on search termsI started messing about with sourcing data from twitter, looking to use this with <a href="http://nltk.org/" target="_blank">NLTK</a> and maybe <a href="http://lucene.apache.org/solr/" target="_blank">SOLR</a> sometime in the future. I created a simple iPython Notebook on how go grab data from a twitter search stream, all details included in the notebook<br />
<br />
I unfortunately couldn't find a simple way to imbed the notebook in Blogger, not wanting to waste time on that I just hosted it as a Gist. It can be viewed here: <a href="http://nbviewer.ipython.org/80b51b024d90d5989189" target="_blank">NBViewer</a>Brianhttp://www.blogger.com/profile/09252223334818664049noreply@blogger.com4tag:blogger.com,1999:blog-2976711854892236281.post-5080696365367613942013-09-04T21:41:00.000+02:002013-09-04T21:41:26.389+02:00Review: Learning IPython for Interactive Computing and Data VisualizationI have just completed working through <a href="http://www.amazon.com/Learning-Interactive-Computing-Visualization-ebook/dp/B00CITNPHQ/ref=tmm_kin_title_0?_encoding=UTF8&sr=&qid=" target="_blank">Learning IPython for Interactive Computing and Visualization</a>,<br />
<br />
Having seen references to <a href="http://ipython.org/" target="_blank">iPython</a> from my first ever google for 'python', I somehow managed to disregarded it with the sentiment of who works in a console?? or a browser notebook? what is that? ...<br />
I need an IDE with folders / modules / files / projects... what a shame I wasted so much time...<br />
I blame too many years in Visual Studio, Eclipse, Jetbrains IDEs and XCode for making me ignore this long.<br />
Thankfully I have gotten past that, and this book helps you getting there fast... < 150 pages fast.<br />
<br />
IPython, and especially the IPython Notebooks are great tools. I can see it being awesome for a whole number of tasks:<br />
<br />
<ul>
<li>learning python and working through books and tutorials</li>
<li>running data mining brainstorming sessions </li>
<li>showing people the latest and greatest stuff you've have come up</li>
<li>quick cython implementations & performance experiments</li>
<li>processing multiple cores / servers </li>
<li>I even saw Harvard now uses it for <a href="http://nbviewer.ipython.org/urls/raw.github.com/cs109/content/master/HW0.ipynb" target="_blank">homework</a> assignments.</li>
</ul>
<br />
That list can just go on and on, but coming back to the book. It was targeted at 2.7, obviously I didn't listen and worked through it in Pythong 3.3., but thankfully there were only a couple very minor changes:<br />
<br />
The book uses urllib2 in a couple, that can be replaced with:<br />
<br />
<i><b>import urllib</b></i><br />
<i><b>r = urllib.request.urlopen('<url>)</url></b></i><br />
<br />
<br />
For the networkx example where was also a slight change:<br />
<br />
<i><b>sg = nx.connected_component_subgraphs(g)</b></i><br />
<br />
This returned a list of graphs, not a graph, so I just looped the following:<br />
<br />
<i><b>for grp in sg:</b></i><br />
<i><b> nx.draw_networkx(grp, node_size...</b></i><br />
<br />
<br />
Then for the maps exercise I did not have all the dependancies:<br />
I need to Install <a href="http://trac.osgeo.org/geos/" target="_blank">GEOS</a>...I used MacPorts for that:<br />
<b><i>sudo port install geos</i></b><br />
<br />
Then in my .bash_profile I added:<br />
<b><i>export GEOS_DIR=/opt/local</i></b><br />
<br />
To refresh the profile:<br />
<b>source ~/.bash_profile</b><br />
<br />
Then for Basemap, downloaded the zip, <a href="http://matplotlib.org/basemap/users/download.html" target="_blank">here</a>.<br />
Followed by(in basemap-1.0.7 dir):<br />
<b><i>python setup.py install</i></b><br />
<br />
That's about it, concise intro for a great product.<br />
<br />
Now to really put it to the test the next book I am working through:<br />
<a href="http://www.amazon.com/Building-Machine-Learning-Systems-ebook/dp/B00E7NC9D2/ref=sr_1_1_bnp_1_kin?ie=UTF8&qid=1378322976" target="_blank">Building Machine Learning Systems with Python</a><br />
<br />
<br />Brianhttp://www.blogger.com/profile/09252223334818664049noreply@blogger.com4tag:blogger.com,1999:blog-2976711854892236281.post-12425345152418176362013-08-25T19:23:00.002+02:002013-08-25T19:23:28.609+02:00Things I learned while reading Programming Collective Intelligence.I have been working through <a href="http://www.amazon.com/Programming-Collective-Intelligence-Building-Applications/dp/0596529325" target="_blank">Programming Collective Intelligence</a> over the last couple months. I have to say it's probably been one of the best learning experiences I have had in my years programming. Comparing to some of my previous technology stack / paradigm change experiences:<br />
<a href="http://en.wikipedia.org/wiki/Muggle" target="_blank">Muggle</a> -> VB4<br />
VB6 - > Java<br />
Java -> .Net<br />
Java -> iOS mobile / game development<br />
This is the biggest, not so much just from the technology stack, but more purely due to the size and complexity of all things ML, AI. Not coming from a mathematical / statistical background, it's really quite a deep hole to jump into, and quite a challenge.<br />
<br />
Not only did this book walk me through a bunch of machine learning and data analysis theory, it got me to learn Python and in translating to Java I also got introduced to a whole bunch on Java related tools and frameworks.<br />
<br />
I created blog posts for chapters 2-8, and decided to just work through the Python for chapters 9, 10, 11 and 12, for 2 reasons;<br />
1. Improve my Python<br />
2. Get it done so I can move onto my new personal project, using all this ML and Python knowledge to create an cross platform application with a rich UI using either Kivy or QT.<br />
<br />
To list some the ML / Data Analysis topics covered in PCI:<br />
<br />
<ul>
<li style="margin: 0px; text-align: left; word-wrap: break-word;">Classifiers </li>
<li style="margin: 0px; text-align: left; word-wrap: break-word;">Neural Networks</li>
<li style="margin: 0px; text-align: left; word-wrap: break-word;">Clustering</li>
<li style="margin: 0px; text-align: left; word-wrap: break-word;">Web crawlers </li>
<li style="margin: 0px; text-align: left; word-wrap: break-word;">Data indexers </li>
<li style="margin: 0px; text-align: left; word-wrap: break-word;">PageRank algorithm </li>
<li style="margin: 0px; text-align: left; word-wrap: break-word;">Genetic Algorithms</li>
<li style="margin: 0px; text-align: left; word-wrap: break-word;">Simulated Annealing</li>
<li style="margin: 0px; text-align: left; word-wrap: break-word;">K-Nearest Neighbours</li>
<li style="margin: 0px; text-align: left; word-wrap: break-word;">Bayesian filtering</li>
<li style="margin: 0px; text-align: left; word-wrap: break-word;">Decision trees </li>
<li style="margin: 0px; text-align: left; word-wrap: break-word;">Support vector machines</li>
<li style="margin: 0px; text-align: left; word-wrap: break-word;">Kernel Methods</li>
<li style="margin: 0px; text-align: left; word-wrap: break-word;">Linear Regression</li>
<li style="margin: 0px; text-align: left; word-wrap: break-word;">Evolving intelligence </li>
</ul>
<br />
<br />
<br />
The Java tools, libs and frameworks investigated:<br />
<br />
<ul>
<li><a href="http://www.heatonresearch.com/encog" target="_blank">Encog</a></li>
<li><a href="http://www.neo4j.org/" target="_blank">Neo4J</a></li>
<li><a href="http://code.google.com/p/guava-libraries/" target="_blank">Google Guava</a></li>
<li>Crawler4J</li>
<li>Java Tuples</li>
<li><a href="http://graphstream-project.org/" target="_blank">Graphstream</a></li>
<li>SQLite</li>
<li>Rome</li>
<li>JSoup</li>
</ul>
<br />
<br />
Python tools, libs and resources discovered:<br />
<br />
<ul>
<li><a href="http://kivy.org/#home" target="_blank">Kivy</a></li>
<li><a href="http://www.jetbrains.com/pycharm/" target="_blank">PyCharm</a></li>
<li><a href="http://qt-project.org/" target="_blank">QT</a></li>
<li><a href="http://docs.python-guide.org/en/latest/" target="_blank">The hitchhikers guide to Python</a></li>
<li><a href="http://ipython.org/" target="_blank">IPython</a></li>
<li><a href="http://matplotlib.org/" target="_blank">MatPlotLib</a></li>
</ul>
<br />
<br />Brianhttp://www.blogger.com/profile/09252223334818664049noreply@blogger.com7