Selecting a new programming language to learn.

I have been itching to learn a new language, but being a Java freak, I always end up convincing myself to spend the time and effort discovering, investigating or playing with something in the Java open source stable, Spring, Hadoop, Joda Time, Hibernate, Maven, Hazelcast, EhCache etc etc. Developing in Java these days is almost purely about knowing and wiring together frameworks, which is both a good and a bad thing. (as well as a topic for another day).

Now to get myself to not redirect the "new language" energy into Y.A.F (yet another framework) I decided to give the languages out there a proper look and see which would be the best fit and most beneficial to my work, marketability and just general 'IT Zen'.

So what do I require from a language:
IDE... my number 1 thing is an IDE, if there isn't a decent IDE for a language it is frankly not worth the time and effort. I don't see myself as a "scientist" where I feel the need to cause myself pain and inconvenience to be "pure". I want a comfortable productive working environment, and VI or Notepad with a command line utility ain't it.

Established... Every couple years someone somewhere tries to define some new language, and most of those die in obscurity for example brainf*** or anything listed on Esolang.

Popular / In Demand... As with most things, popularity is good, it means:
open source community, support and most importantly jobs. If you ever want to see the current popularity of a language Tiobe is the site to visit.

So who are the contenders out there?
Based on Feb 2011 Tiobe index:
Java is still the no.1 most popular, it has awesome IDEs and it's been around for just more than 15 years (January 23, 1996), but thankfully I know Java reasonably well :)... so moving right along... To narrow down the list quickly, I won't look at any languages that are losing popularity, for obvious reasons, so from the top 20 on the Tiobe list that excludes: C, C++, PHP, VB, JS, Perl, Ruby, Delphi, Go.
(C, C++, PHP, VB, JS, Perl, Ruby, Delphi, Go.)

Which leaves behind:
Python, C#, Objective-C, Lisp, NXT-G, Ada, Pascal, Lua, RPG

Now there is a line between established and old, I am going to make a call that could offend some people and say Pascal and RPG are just old. (Pascal, RPG)

Ada, don't know much about it, after reading the ADA overview, it seems okay, going to exclude it based on popularity. (Ada)

Lua, from a quick read it is a scripting language. (Lua)

NXT-G has something to do with lego or some robotics, not very mainstream. (NXT-G)

Lisp again like Ada, at first glace seems fine, just not popular enough. (Lisp)

Then there are the "New, built on other platforms" functional languages: Scala, F#, Clojure. Although very temping being on the bleeding edge, it's not all that profitable or marketable yet. I'll give them some time to standardize, settle down and see if they are widely adopted. They do appeal greatly to my inner geek, so will always be keeping an eye on them.

So this leaves me with:
Python, C#, Objective-C, (and Java).

Straight away based on the above list we can Tick: IDE, Established and Popular / In demand. We all know they have decent IDEs: Eclipse, XCode, Visual Studio, (IntelliJ and Netbeans). They have also been around and are well known.

Now looking at number of jobs:
Found a site (Simply hired) with a graph displays the percentage of jobs with your search terms anywhere in the job listing. Since June 2009, the following has occurred:

Python jobs increased 72%
C# jobs increased 77%
Objective-c jobs increased 268%
Java jobs increased 76%

Python, C#, Objective-c, Java trends

Python jobs | C# jobs | Objective-c jobs | Java jobs

With the recent boom of iPads and iPhones the Objective-C percentage is not all that surprising. I do have a problem with Apple, Objective-C and XCode and that problem is you need a Mac to run it. Once you start down that road you end up having to change everything to Apple, and I am not ready to do that. So for now I am going to drop Objective-C out of the running. Although if I ever do buy into the whole Apple thing, this will got back to the list.

Leaving me with Python and C#, looking at their salaries compared with Java:
(Data from Payscale).
US Data
Java
PayScale - Java Skill Salary, Average Salaries by Years Experience

Median Salary by Years Experience - Skill: Java (United States)

CAREER TOOLS: Salary Calculator, Career Path Tool, Cost of Living Calculator, Meeting Miser

Python
PayScale - Python Skill Salary, Average Salaries by Years Experience

Median Salary by Years Experience - Skill: Python (United States)

CAREER TOOLS: Salary Calculator, Career Path Tool, Cost of Living Calculator, Meeting Miser

C#
PayScale - C# Skill Salary, Average Salaries by Years Experience

Median Salary by Years Experience - Skill: C# (United States)

CAREER TOOLS: Salary Calculator, Career Path Tool, Cost of Living Calculator, Meeting Miser

South Africa Data

Java
PayScale - Java Skill Salary, Average Salaries by Years Experience

Median Salary by Years Experience - Skill: Java (South Africa)

CAREER TOOLS: Salary Calculator, Career Path Tool, Cost of Living Calculator, Meeting Miser

Python
PayScale - Python Skill Salary, Average Salaries by Years Experience

Median Salary by Years Experience - Skill: Python (South Africa)

CAREER TOOLS: Salary Calculator, Career Path Tool, Cost of Living Calculator, Meeting Miser

C#
PayScale - C# Skill Salary, Average Salaries by Years Experience

Median Salary by Years Experience - Skill: C# (South Africa)

CAREER TOOLS: Salary Calculator, Career Path Tool, Cost of Living Calculator, Meeting Miser

Based on the US data, I would have gone with Python, it's not as popular as C# but the pay is slightly better, I would also get to keep using Eclipse (PyDev) and Spring, but as soon as I looked at the South African data, I realized something, Python is really not big here. I manually went searching for Python positions advertised.. and found a grand total of 2 and the salaries were not good.

(Python)

Leaving C# as the last language standing.

It's got Visual Studio (even a free version Visual Studio Express), It has proven itself over the last couple years, it's out innovating Java at the moment, there's a ton of jobs, a whole range of certifications and the salaries have closed the gap on Java.
Seems quite a logical choice to me.

To top it off, I have also used C# many years back, so it won't not entirely new. Most of the successful Java open source projects (Spring, Hibernate etc etc) have been ported so all that knowledge is reusable, which also counted a little in my decision. Now I just need to stop working 12-14 hours a day, and I can focus on getting back to my Microsoft roots with little C# as a Java developer. Hopefully a couple months after that I can go through this process again, looking at Python, Objective-C, the mobile platforms (iOS, Android, windows) or maybe rather a concept change to functional with the likes of Clojure or Scala.

Validate XML against its XSD

Just a quick code snippet for possible future reference. How to validate a XML against it's XSD.

Surviving the Wild West Development Process

Now in a perfect world, we all strive for a some kind of development process, be that: Waterfall development, Prototyping, all things Agile or whatever process the CIO / CTO /CEO got sold on by some consultancy.

But in the real world what sometimes ("sometimes" being quite often actually) happens is:
What I like to call the Wild West Development Process (WWDP)... the main principle is basically “promise first and ask questions later”.

Getting started with WWDP is pretty simple:
1. You need someone, somewhere in a place of power and influence in your organization to promise software by some unachievable date.
2. Add in a eager marketing department and some press releases…
3. stir and leave to simmer.
4. After simmering for a while inform the software development department that they must have something, that does "stuff" and the release date can not change.

I am not sure if this is a result of a immature IT industry in my country, but I have spent many years in environments like this. It even seems to find me in the most unlikely employers, the large corporate organizations like insurance and banking institutions, where general red tape and corporate processes should limit this. I started my career in one of those “startup-sell-information-tech-and-business-processes-to-anyone-cheaper-than -anyone-else-type-consultancies". Which means every project was driven by WWDP so I started in the deep end, and for about 4 years I knew no other way of developing software. I currently find myself in another one of those projects. I'd like to share some points that I found helpful in actually delivering some software in unimaginable unmovable deadlines.
(to be fair some of these are in the standard development processes mentioned earlier, maybe with just different priorities)

Pick the right developers
Not everyone can cope in this environment, knowing your staff / team is very important: late nights, constant crisis, constant changes and bucket loads of pressure and stress aren't for everyone. Some people thrive, some survive and run for the hills after, some just end up whimpering in the corner and need to be replaced (which is a disaster).

Have the developers involved.
Projects like these don't allow for the usual requirements / design / develop / test paradigm, all of these functions run concurrently. Having the developers know as much detail as the architect, analysts and testers is vital. Developers will have to make decisions affecting the architecture and business functionality constantly, if everyone has the same picture you'll save time.

"Prototype" early
I say "Prototype" in quotes because unlike an actual prototype there won't be time to throw this code away. This "prototype" is to have the system in it's entirety up and running very early in the project. It can be a "shell" but having it up means testing can start early even if it's just the basics and no business functionality. Since there are no real requirements, make sure the design caters for adding them as they are defined.

Integrate early
If the system has any integration points, between systems, teams or external parties, make sure this is also in the "Prototype Shell", integration always takes longer than you think, having the interfaces up and available to test is crucial.

Keep track of communication
When the shock of not meeting an unachievable deadline kicks in, someone will always look for a scapegoat, always keep at least an email trail making sure you aren't that goat.

Automated builds and deploys
The standard agile practice of continuous integration is as important as ever, and even if you skip other parts of you software development cycle this is something that should not be dismissed.

Focus
With hundreds of things that need to happen a matter of days or weeks, it's very easy to try do 6 things at a time. Don't. Micro manage your time and tasks and focus. It's easy for us developers to start a task, see something, change something and lose a day which you don't have in the first place. If you have to, have someone keep an extensive list of "TODOs" that can be used in the cleanup phase.

Clean up
Once the unachievable deadline has been met, before starting the next phase, have at least one "cleanup" release. This cleanup phase should add very little or preferably no new functionality, but will allow for design and development decisions made in haste to be reviewed and refactored.

Pace Yourself
No matter how super human you may think you are, all of us have a limit to how many hours we can keep going. At some point you are actually being counter productive and a couple more hours sleep will actually help the project a lot more.
Know your limits.

I am sure there are a couple more, but currently being on a WWDP project I need to re-focus and leave that for the clean up phase later :)