His recent blog post on GeoFlow (an add-in for Excel that delivers 3D visualisation and story telling) has raised the issue of whether MVP’s should criticise Microsoft.

I think its fair to say that the consensus from respondents in the blog comments, is yes criticise if its warranted, but don’t get into the habit of always moaning. Positive feedback is as crucial as negative feedback. You could also argue negative feedback can have a positive effect if handled appropriately (both the giving and receiving).

I should also point out that the suggestion that MVP’s should not criticise Microsoft  (that I saw) came from another MVP (not an employee of MS, I assume). Please read the blog post and comments if you want to be sure as there are many and I skimmed some of the entries.

Many of the comments were from IT professionals trying to deliver mobile BI solutions to their organisation or clients and were expressing their frustration at the lack of Microsoft mobile BI support. Jamie’s blog discusses the lack of integration of the GeoFlow project offerings into the core MS BI delivery platform.

I expect every MVP at times, will question if the role is good for them. I guess it is very beneficial, for mostly intangible reasons. I imagine being a good MVP takes a huge amount of time and personal cost to attend conferences, user groups etc. and reasons to accept or decline the role would take into account many more things than the freedom to criticise Microsoft.

What about biting the hand of your employer or line manager, which brings me to the other side of the coin, perhaps kissing the hand that feeds you?

Within an IT team or department, the trouble with always moaning is you can become a miserable git, stop enjoying your work and more importantly, be too busy whinging and miss opportunities to deal with the initial cause of the moaning. This is crudely referred to as ‘having your head up your …”. No-one will want to talk to you, work with you and you can forget anyone listening to you. Even when you are right. Think Peter and the Wolf.

But the opposite can be just as unproductive.

Beginning of the week – “Any issues with meeting the deadline for your [insert project deliverable] ?” asked the project manager.

“No its all good” answered the programmer.

End of the month “why is your [project deliverable] not ready?” asked the project manager.

“Because [insert excuse]” said the programmer.

The excuses are mostly irrelevant at this stage for the project manager. Project managers [should] want to intervene and resolve issues before deadlines are missed. Firefighting when its too late is okay every now and then but it becomes soul destroying when its endemic. I bet it’s often at the heart of good IT professionals moving onto to greener pastures.

Never speaking out could be considered cruising along on Prozac or sailing with the “it’s all good” and “have a great day” crowd. Or maybe its a stubborn streak, preventing you from admitting that something is defeating you. Perhaps you are in danger of being labelled “doesn’t play well with others”?  Maybe you are not at your best at the moment and need a break?

Or maybe you dont think its your place to question the decisions made by your boss?

I strongly believe that if you think a directive handed down by your team lead, project manager or IT boss is wrong (good luck if all three of these lines can directly request you to do tasks!!!), or misses something, you should challenge it in a respectful way at the right time. Its also the best way to learn. If you happen to be wrong or perhaps didn’t fully understand the requirement etc. a good team lead or architect will take the time to explain and get you up to speed. If you aren’t brave enough to speak up and admit you don’t understand or question a proposal or task, you wont learn.

Maybe you aren’t wrong. Often there is no black or white answer and every idea should get consideration (maybe 30 seconds perhaps 30 minutes). Managers and design leads / architects aren’t perfect. They (should) make considered use of the team they are working with to collaborate and produce robust solutions. The end result of the combined thinking of everybody who can contribute will generally be better than the work of an individual.

However, it is their job to lead and propose solutions. But as I said they aren’t perfect. So by sitting on your hands and never challenging those above you in hierarchy, experience, competence or pecking order, you are doing yourself, them and your employer a dis-service.

Sadly, some managers lack the experience or humility to permit “underlings” to contribute to the overall decision making process of a department. These types can be identified by the habits of not conducting meetings with more than 1 person (maybe 2 if pushed), having favourites amongst their line managers (who often turn out to be ‘yes’ people or are playing the long game and dont rock the boat). They usually don’t respect lines of authority (ie. go around your team leader with requests) and they are often micro-managers and cant delegate.

It is generally very difficult to find opportunties to offer constructive criticism in these types of environments. Maybe you will be lucky to  be working within a tight team and your team lead or line manager shields you from these issues.

So if you have a supportive workplace, where top down, bottom up collaboration is encouraged, then work with it and find solutions to vendor shortcomings. Get hold of the GeoFlow add-in and innovate. I am sure thats what many will get on and do.

However, if the traits I described ring bells for you and you are feeling frustrated or stifled then you need to weigh up your options. Maybe you are considering going around your direct report to offer constructive criticism up the line. If so, its probably already too late and you should be planning your next career move. Your job might offer security and stability, but being a down trodden miserable or stressed IT professional isn’t going to make you happy and wont produce good code in the long run.

Above all, don’t sit on your hands and moan in the tea room (or even worse, to your significant other).

Get involved and contribute.

The free event, put on by the SQLFAQ group, sold out two weeks prior to the date. A sure sign of the increasing popularity and usage of Microsoft SQL Server.

Here is the slide deck from my presentation. Please feel free to contact me if you have any questions.

If you weren’t able to attend, it covers the evolution of the relational database model and the major players, what a normalised data model looks like and how a star schema database deliberately moves away from that OLTP optimsed design, to deliver the basis for high performance decision support solutions.

I was completely blown away by the packed room of approx 125 attendees. Thanks for listening. I really enjoyed myself and I hope you found the time you gave up worthwhile and a big thank you to Tony Rogerson and the rest of the SQLFAQ team for the opportunity.

International Rescue might have started out as a small family business with ‘Father’ orchestrating operations using instinct & raw intelligence, but it would soon have developed into a considerable enterprise with R&D, product development, branding and a network of contacts across the globe.

Stupid fantasy question? Perhaps it’s not so different from many of the growing enterprises across the globe. I think many businesses struggle to transition from small business to a corporate enterprise and the transition to more formal decision making processes is key challenge. Typically, directors are likely to still be using experience, instinct and if they have come to grips with delegation, a core of able lieutenants keeping operations and logistics finely tuned.

When it comes to ‘business intelligence’, they are likely to be struggling to shift from traditional reporting deployments (providing data on what happened) to fact based decision support solutions, driving real time KPI monitoring. Forget analytics driven insight. It’s tough enough responding to market conditions and I bet many financial controllers are still chasing that dream of daily profitability KPI’s.

And it not just a problem for new growing companies. Older, established enterprises with legacy computer systems also face similar challenges. Many production capture systems are still data silos that impede decision making because of the legacy host. Lets not forget many of the features of relational & and other computation theory have only recently become commercially viable due to the advances in solid state memory.

Therefore, I think ‘Big data’ is so far down the road for most companies its got that ‘oasis’ heat shimmer thing going on. The hype that is currently being generated makes for a lot of FOMO (fear of missing out). This makes these companies a prime target for ‘marketing’ coyotes & foxes and they are in plague proportions. Dont get me wrong. Big data is real, its just not the first step on the journey of analytics driven decision making.

Companies such as Google, Facebook and Twitter (to name a few) are inherently ‘big data’ concerns. It doesn’t follow that every other company would benefit from ‘getting on board’ this revolution. However, I firmly believe that all businesses would benefit from improved decision support solutions. Its a matter of choosing something appropriate that can scale as you grow.

So if you have been “tuning in” to this new IT drama series on the web, called “BIG data” and are getting a bit tired of it, you will be pleased to know that there is now a spin off series called “the data scientist”.

So in addition to our daily staple diet of yet another marketing led big data offering, we will very soon see a flood of recruitment agents touting a stable of “data scientists” on their books or similarly a swathe of professionals changing their job titles on LinkedIn.

But hang on a minute, didn’t Howard Dresner recently tell us that the capabilities of a data scientist are extremely rare. Those capabilities apparently being “someone who provides the capability to increase the value of an organization’s information assets”.

Too right he did. However, if the Twitter comments of some fellow SQL professionals who have been recruiting for skilled T-SQL, DBA & ETL talent recently are indicative, then there will be plenty seeking work out there, who are prepared to bluff their way into this role. Many agents will struggle to filter them out as the IT industry itself comes to grips with understanding this new analytical direction.

Anyway Howard hosts a weekly BI twitter session and I might start joining in. Please check out his original article that I am quoting to get the full story.

The Register.co.uk reported a few weeks ago about predictions of a global skills shortages citing the McKinsey Global Institute report. The author, Mark Whitehorn, suggests a data scientist would excel at analysing data, particularly large amounts of data that does not fit easily into tabular structures, so called “big data”. Sounds quite reasonable to me. I am a long time reader of the Register and recommend it highly to anyone in the tech industry. They give ‘Fleet Street’ a run for their money on inventive headlines.

In another report, Bruno Aziza says he has found the most successful data scientists balance brains and brawn–they can manipulate databases like its nothing, work well with other coworkers and present findings to the executive board. “Think of a data scientist more like the business analyst-plus,” says Aziza.

In the same report Jeff Hammerbacher, who previously led the data science team at Facebook, recounted that his team would use Python, R, and Hadoop, and then have to relay the findings to a non-technical team on any given day. The more you know, the better you’ll be prepared to solve the day’s problems.

I do wonder what Jeff meant by “non-technical”. Given the specialisation that goes on in IT these days, that could mean the front line data analysts that support the business report writers. Or it could be the analyst in Marketing who is planning the next seasons promotions. Perhaps its director level strategic planning sessions. It does highlight that converting data into information is still a classical task that requires directed effort that is linked all the way from senior decision makers through to operational users. There is no straightforward hierarchy in this process as data and information flows are very intertwined, now more than ever.

I have only recently subscribed to the Smart Data Collective and follow them on Twitter (@SmartDataCo). There are some very clever, well informed contributors on this site, as well as curation of high quality vendor sponsored whitepapers.

A recent Gartner research note has injected a healthy dose of “a warp drive plasma” equivalent into the “big data” mix. The headline quote for big data (but by no means the only interesting one), is “By 2015, big data demand will reach 4.4 million jobs globally, but only one-third of those jobs will be filled.” Another significant trend was that CMO’s (chief marketing officer) will begin to take over the budget that CIO’s are used to controlling. Now that would be interesting.

So, clearly interesting times are ahead. Gartner think Marketing will take control of a huge chunk of what was traditional IT budget. Cloud based PaaS, SaaS & IaaS are making it even easier for business units to go around IT department backlogs and unrealistic timelines. There are some very innovative cloud solutions out there and it certainly seems the way of future IT, but its not necessarily right for your business, or that specific requirement, now or ever. Gartner also think there wont be enough talent to meet demand.

I advocate loads of caution and pragmatism when getting out your cheque book for a third party service or solution and also if you decide to embark upon this in house. If you wish to better understand the roots of ‘big data’ and get a handle on what its really about, then I would advise starting with the book “Analytics at Work, Smarter Decisions Better Results”.

In the opening paragraphs this line struck a chord with me – “Rigorous analytical testing can establish whether your intervention [ … a customer communication, product discount, sales promotion, product placement, loyalty scheme … ] is causing desired changes in your business, or whether its simply the result of random statistical fluctuations”.

If you are part of a company that is facing the challenges that I have referred to, then this book is an excellent starting point. I think its an especially useful tool for understanding & communicating core issues and concepts to those within an organisation not familiar with analytics. However, its primarily presented as a methodology and this is where I found it most useful. There are no shortage of excellent tech manuals on how to get the most of database engines and how to build efficient ETL pipelines, which coupled with experience, whether personal or shared within a team, can deliver top quality decision support solutions.

However, a well conceived and robust analytics strategy can focus a business on what is really important right now, and ensure that tactical endeavours have the greatest chance of success. The right answers to the wrong questions, delivered in near realtime don’t help. I think this is why many of us IT techies have an affinity with the answer “42″, as we cynically appreciate the importance of asking the right question. The task of formulating the right questions is a business wide task requiring strong leadership, delegation and empowerment. This is also why I strongly believe that steering of BI / decision support solutions must be lifted out of IT, as IT types aren’t traditionally the best communicators and generally don’t have a business wide perspective or remit.

An alternative to being drawn into the “BIG data” hype, is to start a business intelligence unit, with a reporting line and budget separate from or in tandem with the IT department (you cant do it without them).

I would begin with a trinity of one finance domain expert, one marketing domain expert (or supply chain or analyst representing your core business activity) and a good data analyst from IT (or failing that an Excel expert from within the business). With a remit from and reporting to the relevant departmental directors (or board), define some useful, tangible questions & outcomes, with a short timeframe (weeks) and get started. Treat it like an old fashioned science project, predict the outcomes and learn from the results, expected or otherwise.

But don’t mistake this for a purely technical excise. Creation of an independent BI unit will also potentially shift the control of information within the business and as Francis Bacon said, “Knowledge is power”. The role of those steering the BI unit is crucial and its never easy to implement change within an organisation. Another challenge for those in charge of a business is to accept the reality of solid analysis, especially when it flies in the face of what the collective opinion currently is.

Seek out expertise in-house if you have it. Otherwise, hire an experienced BI consultant or data architect who has a track record and can demonstrate competency. Hiring a full time ‘data scientist’ or even a rockstar ‘data analyst’ and setting them loose from the outset, is effectively getting your tools out before you have a design or plan. Also, if this is new ground for your company, then you may not have the qualifications to hire the right skills and know when they are needed.

This article arrived as I was working on the final draft and is very timely. With a title of “The End of the Expert: Why No One in Marketing Knows What They’re Doing” its quite controversial, but makes some excellent points. This change of landscape could happen in any area of your business. Dont leave it too late to change direction. Robust BI solutions can provide very early warnings and analytics will help your company work out where to head next.

The latest debate is that Hadoop and other no SQL solutions will bring about the death of RDBMS and the EDW. OODBMS was going to kill RDBMS ten years ago. Didn’t happen and it won’t happen this time.

Purposeful analytics solutions are shaping up to be very powerful tools, but I think right now, a CIO would be crazy to embark upon a program of shifting the corporate data asset out of proven RDBMS technology platforms, underpinned by an experienced and populous talent base, into the brave new world overseen by this new breed of data scientist. It needs time to mature and even then I am not so sure it is a one size fits all solution. EDW methodology is as old as relational theory and is still relevant.

What also concerns me is that government & retail marketing seem to be at the head of this near hysterical big data push. This combination is worrying and I am concerned about the potential standards and ethics behind some of the data mining initiatives being conceived and implemented. This ‘BIG data’ push is coming at a time when existing protection instruments are coming apart at the seams, due to being conceived so long ago and no longer being fit for purpose.

Finally, I recall reading an article in one of the Computer magazines over a decade ago, which advised being wary of ’silver bullet” solutions that promise the world. Usually to be delivered in a guaranteed time frame and they will solve all your problems (usually sold by someone in a teflon coated suit). BE WARY. There are plenty of foxes & coyotes out there. I think the recent bad news for HP highlight this is still very relevant today.

FAB Virgil

The tagline of the presentation was “the core principles of the Star Schema database, used for decision support solutions and as the bedrock for Analysis Services cubes and the new Tabular model”.

It covers the evolution of the relational database model and the major players, what a normalised data model looks like and how a star schema database deliberately moves away from that OLTP optimsed design, to deliver the basis for high performance decision support solutions.

I cover the basic features of the star schema database, dimension and fact table examples and considerations for best use of these database designs.

If you are starting out in this field then I hope this presentation provides some context and useful tips. I extensively used Wikipedia for my research and have provided loads of links. So whenever you want to find out when Larry Ellison spotted an opportunity waiting to be exploited, which RDBMS system spawned many others and who is Jimmy Treybig, this primer brings many of these trivial pursuit answers for data professional together in one place.

Click to view the Apple Keynote presentation in your browser.

PS. if you spot any errors, or think I have done someone or some company a disfavour by omission, please let me know.

Here is my ideal kit list;

• MacBook Pro 15″
• SSD or Momentus hybrid drive
• Optional conversion of DVD bay to a second HD bay
• Apple Thunderbolt Display (or similar external monitor)

The software that you will need does depend on whether you are operating as an independent consultant or whether you are working within a coproate domain environment. The main difference stemming from this will be the software you personally need to deploy and it may not differ.

To run Windows you will require a copy of VMWare Fusion 4 (version 5 is now out). This runs as a native 64bit OS X application. I have tried Parallels (an early version and version 6) and I found it very poor for setup and the support was sub standard.

You absolutley need loads of RAM and fast disk storage. I think a minimum of 8gb of RAM is essential if you are too get the best of both operating sytems. These MBP’s run 16gb.

I think a mobile computing device is essential these days and the MacBook Air is not really designed for user upgrading, so the MacBook Pro is the obvious choice. However, I started writing this piece before the mid 2012 hardware refresh and there are now some alternatives.

The Macbook Air, configured with 8gb of RAM and 512gb of SSD is sufficiently configured to run a Windows VM with plenty of on board storage. The 2.0ghz i7 CPU packs plenty of power. The Macbook Pro 13“ isnt much different from the Macbook Air other than a faster CPU and the option for a traditional HD. However the 15” MacBook Pro has the separate graphics card (now from NVIDIA). This could make a difference for graphics intensive work, potentially making a difference with SSIS, however I can be certain of that. I do know that software like Pro Tools (music creation) suffers from the lack of a distinct graphics card, where heavy compute for maths operations plus simultaenous screen painting strains the system.

In the end I still recommend the Macbook Pro 15″ (non retina simply due to the excessive price and lack of user upgradeable options).

The best new feature with the mid 2012 refresh is the inclusion of USB 3. This is a super fast interface, much better than Firewire 800 and not far off Thunderbolt. Also, you can buy a very decent 320gb drive for under £60. I did a copy using CCC (Carbon Copy Cloner) from my SSD drive to the USB 3 drive of a 40gb Windows VM file. The first gigabyte was done in 20 seconds. it completed the 40gb in 10 minutes.

I have also tested running the Windows 7 VM machine straight off the USB 3 drive and it started up in an adequate time. Certainly not as fast as my onboard OCZ Vertex 4 SSD but very good when compared with a stock onboard 7200rpm HD. This is certainly and excellent option if you dont want to do any DIY work replacing your DVD drive with a HD / SSD caddy.

The biggest risk using Windows is Windows OS corruption. This is even more likely when using virtual machines. I have had instances where the VM has become corrupted and I have had to use a backup. However, beware if your VM is attached to a Windows domain as if the backup is too old, you can get a ‘untrusted’ error message and recovery from that wasn’t possible.

So I have two uses for Carbon Copy Cloner and external drives. I am now using a G-Tech Firewire 800 1tb external drive to contain a clone of my entire system drive. This is scheduled to run every night in the wee hours. So if I am at a customer site and I get the spinning wheel of death (the OS X equivalent of the Windows BSOD) then I can boot the system from the external drive. This means no loss of time diagnosing problems and rebuilding the system. This drive usually stays in my bag as this type of issue is a rare occurence.

Next installment – Integration and software options

At the time a city analyst declared the group’s finance department ‘not fit for purpose’. Its shares tumbled 38% on this news. A month ago, the group announced the recruitment of a new COO (new role), a new CFO and a new Head of Retail in an effort to boost the management team and increase the professionalism of the group.

In general terms IT provisioning for a business such as SuperGroup falls into two broad categories. The first are the type of solutions that support the procurement and sale of goods, wholesale or retail. This is traditional IT and it has been around for a long time, delivering improvements and cost savings in areas like HR, Supply Chain, Marketing etc. New sales channels and models have developed over time (eg. the web, eBay etc), but the focus is still on response times, availability and scalability. It’s all about getting the job done, modeling business processes and providing automation and capture systems as appropriate.

The second category of IT provisioning is centred on decision making. Business relies very heavily on information but not all are fortunate enough to have timely and reliable data to work with. Manual number crunching and intuition (“gut feel”) have served many a business well but it’s not enough anymore, especially as a business grows beyond the point where everybody knows each other by first name.

Traditionally, information for decision making in larger businesses was delivered either via the core ‘online’ system or through nightly or cyclic report production runs. Information delivered via online systems generally took the form of single record data items, such as information about the customer, supplier or order that a clerk / call centre operator might be queried about. Reports were run to arrive at the summary totals on the last page, to assess the outcome of the previous day’s business, for order picking, invoicing etc.

Over the last 10-15 years operational data stores and data marts came into vogue with the purpose of feeding derived business intelligence back into the core online systems. Combined with the transition from dumb terminals to PC based client interfaces, IT was able to arm the customer facing areas of the business (and other areas too) with as much relevant information about the “customer” as possible.

Decision support systems began to get airplay in IT departments. Data Services teams were formed to handle the growing number of multi purpose databases and data oriented solutions. The methodologies of data warehousing advocates Ralph Kimball and Bill Inmon, gained significant traction. Programs specific for moving data between disparate systems and databases, got the new moniker Extract, Transform & Load (ETL).

However, still the primary focus of many IT departments was on the core online systems, with a centralised infrastructure and traditional project methodologies. And fair enough too as there has been plenty to contend with. This would be especially true for small, rapidly expanding businesses. However, I am sure there a many exceptions to this generalisation of focus and perhaps some of them are now Fortune 500 companies (and the equivalent across the world markets).

Solutions deployed were quite often ‘brittle’ (a term I heard Matt Benati of Attunity use recently and I think it’s quite apt). That is, not quick to adapt to changes in the business and prone to breaking. An architect I have had the fortune of working with lately has been advocating significant levels of isolation to shield downstream systems (such as a data warehouse) from the impact of change. However, it’s not always easy or practical to do this.

Another new term has been cropping up recently (Attunity again), in the big data discourse. Extract, Expedite and Load (EEL). It’s interesting because the premise here is on getting data out of capture systems in real time and consuming it as fast as possible in the target system. This concept is very compelling. I personally think that it will gain momentum.

The SuperGroup example is a timely reminder that robust BI processes are critical to any organisation operating in todays business environment. A rapidly growing business should treat BI with equal or more importance than core IT systems. Aggressive business expansion without commensurate capability to measure those business activities themselves, is a recipe for disaster. “Reporting” as BI is often lazily categorised, should not be 2nd string. Failure may not be as immediate and obvious as an “offline” order taking system, but it can be just as damaging to the bottom line (or more so).

Larger organisations needs to get tough with in place IT power bases. Core IT systems require stability combined with continuous improvement. BI solutions are a completely different proposition, requiring many traditional IT skills & features. When combined with some of the new amazing new database solutions and storage advances (column based indexing, SSD’s, cloud deployments, CDC software etc), significant improvements can be made.

I am not talking about traditional operational reporting (which can be a deliverable of the BI team). That’s the straightforward, routine, but bottom line piece of the puzzle. BI solutions must also take the tactical and operational goals of the business (derivable from the strategic vision and plans) and provide an integrated view, all the way from executive dashboards, through tactical performance analysis down to operational workbenches and shop floor devices (and all the way back to the top).

However, the often overlooked challenges for BI are the human ones, the process based issues. This links back to how IT systems development needs to embrace and inform business process. Unfortunately this is when resistance to change is encountered, throughout all levels of the business. This was a factor in the 80s and 90s and clearly remains relevant today. Back then the change mainly impacted the front line and more blue collar oriented workers. Now, transitional BI impacts the strategic & tactical decision makers.

There needs to be acknowledgement of the direct and crucial relationship between steering a business through a rapidly changing landscape and the continuous need for timely and accurate data to support the process. If a business is forced to make rapid changes, the supporting data needs to be delivered into the BI system now, sometimes next week, but definitely not next quarter or next year.

This is essential to create the integrated view of business performance as well as ensuring the impact of tactical decisions are measurable in a timely and accurate manner. In other words, a BI decision support system that is directly coupled to the operational systems.

To give a rapidly growing business its best shot at transitioning from “reporting” into a true BI culture, the BI team needs a degree of independence from mainstream IT activities. It may require its own budget & project planning and should report directly into the CxO and / or board of directors. Its role is to serve the entire business management structure directly involved in critical decision making. It is not just another IT system.

So whilst all the buzz is about big data and the cloud, it’s important to keep our eyes on the main goal. Delivery of timely and accurate data into the warehouse, to support business decision making. The new options can enable benefits not possible a couple of years ago, if they are adopted with a coherent business strategy.

I was fortunate to be indoctrinated into the ‘data centric’ view of IT very early in my career. But it has been a backseat for a long time. More of an ‘engine room’ gig. I think in time, the legacy that the current focus on big data and cloud infrastructure movement will deliver, is the re-focusing of the industry onto that most valuable of business assets. Its ‘data’.

As those of us working in IT know, from the outside we are all bunched together, but within the industry the skillsets today are extremely diverse. The pace of change is incredible and organisational change is a given, not something that occurs every now and then.

One thing that remains the same is coding syntax. It seems every year another language is deployed. It all conforms to logic, but blimey it can do your head in.

In the deep past of my career I had experienced significant protectionism of skills outside of work teams but today sharing and mentoring seem to be mainstream.

The IT Crowd, Episode 1, Series 1, featured Moss wearing a custom tshirt with the acronym RTFM. I first heard this one uttered in 1991 by Graham K in response to me asking one to many questions. It expands to Read The F#*k!ng Manual if you weren’t sure.

The second half of my post title, the Software Dummy, came to my attention whilst working on the Y2K bug (yes it really happened and was not a conspiracy). I was part of a team of 4 Tandem SQL / COBOL analysts developing a large star schema decision support system and we were coding for weeks on end. Frequently one of us would sidle over to an associates desk and start describing an intractable problem. So in this scenario you have the coder with the issue they are struggling with and the listener, the “software dummy” (or peer, mentor, sounding board etc).

Now many of you will know that resolving your own software bugs is a tricky business. Helping someone else resolve their software bugs is altogether another thing.

Firstly, you are most probably very busy with your own work. Secondly, even though it’s likely you have similar skills you probably aren’t familiar with the specific business logic or requirement. Thirdly, there is the matter of sustainability; do you catch them the fish or show them how to make their own rod (see RTFM).

What we observed during that Y2K project was the software dummy effect. It’s just human nature to ask someone else for help rather than picking up the manual and working it out yourself. And frankly the manual is often no use when you have a team of similarly experienced programmers and it’s a bug and not a matter of technique.

So what tended to happen is the software dummy would acknowledge the petitioner (the frustrated coder with the software bug) but continue to focus on the monitor in front of them. The coder would continue describing the bug in detail, relaying the events and conditions that have led to the current impasse. The software dummy might occasionally ask a question but usually it was enough to make appropriate noises giving the impression that we were actually listening to the problem description. Invariably we were not. We were trying to hold onto the thread of what we were immersed in.

However, in most cases the coder would have a eureka moment and exclaim “that’s it, I haven’t initialised the host variables for the …” or some other mind numbingly boring detail and return to their desk, grateful for the assistance.

We discussed this as a team and quickly realised we were all guilty of not really listening. However the results spoke for themselves. Clearly the coder with the bug was working out the problem through the process of describing it to someone else. The software dummy was born.

Many years later on and I have to admit I have tried this at home on my wife. I resorted to describing a coding problem to her (she is not an IT person) and she made all the right noises. If I had checked I am sure I would have seen her eyes glazing over, but I was on the hunt for that elusive bug. Eureka, problem solved. Careful with this though. It could shorten your prospects or those of your genetic line.

Back to the 21st century and I found myself software debugging once again. Recently I have begun coding ETL processes using SSIS and adapting Andy Leonard’s (@andyleonard) SSIS framework for deployment at a client site. I was grappling with the subtleties of defining flexible connection managers using expressions & variables and design time versus batch execution time.

Consequently, I had loads of questions. I read the blog posts that Jamie Thompson (@jamiet) and others had made on the topic of parent child operations. However I was really stuck on grabbing parent variable values from the child package. One minute I had it working and the next nada. The cause turned out to be simple but I burnt many hours getting there.

As I was working alone on this and often remotely from my home office, I had no one to bounce my issues off. Enter the internet, forums and bloggers. The support network available now has become distributed across the planet, rather than a bay of office cubicles. It’s very powerful, but at the same time there are new challenges.

Everyone still has their own workloads, but now its very likely your potential software dummys have different employers or work for themselves. @andyleonard & @jamiet were extremely helpful and gave plenty of initial assistance but I could not nail the issue.

Contribution to the SQL community is a mainstay of the SQL MVP ethos. It appears to me that maintaining the balance between supporting the community, reputation, earning money and family time & free time would be very challenging; especially in an age where many people in your network will be working and needing peer support across multiple time zones.

Forums have been around for a long while and I posted my issue on SQL Server Central.com. This is a great resource, but either my problem was too complicated or too obvious for anyone to address.

And I need to apologise to Andy and Jamie here, because it was all to easy for me to fire tweets and DM’s at them when I could not get through this issue.

However, this is where Twitter and to a degree forums have parallels to the software dummy process. I expect Jamie and Andy made appropriate noises whilst getting on with their own business, perhaps thinking, “I hope he works this out soon”. But for me knowing these guys would respond with a quick answer if there was one and the act of writing down the issues, especially the long dialog I had with the sqlservercentral.com forum, helped me work out the problem. It made me review and examine my testing method (remember I had it working and then lost it). It is this methodology of being able to repeat a process in a controlled manner that is essential to debugging.

So whilst the internet and always on exposes the mentor to potentially more and potentially constant interruption, the virtual team relationship can be managed, kept at arms length. The mentor can dip in and out of the work environment to suit their own needs. The coder needing help has a virtual sounding board with many people listening and I am willing to bet that the software dummy effect is alive and well and a member of every software developers team.

The particular vice I refer to is the marvelleous hard disk drive.

They are wonderful things, these addictive objects of our desire. Shiny, varied, formatable, stackable, raidable and available from a multitude of sources at a price range to suit any budget.

However, like any other addiction, the craving for more comes again, usually sooner than one would like.

I remember once upon a time, when big 8″ floppy disks were still in use, a genuine propellor head techie telling me they used to write disk access routines that would cause the big IBM washing machine sized disk cabinets to wobble walk across the computer room floor. What a hoot. Something to do with achieving the right sort of oscillation. I think the capacity of that early commercial spinning platter was 4 meg. I wish I had thought of the IT Crowd TV concept back then. The comedy material has been there for decades.

Nowadays us techie types are counting our storage in terabytes. Its all very justifiable, high speed connected storage vital to back up and maintain our Virtual Machines, video, music & photo collections etc.

If you are a SQL Server DBA you could be talking orders of magnitude more for testing and evaluation, honing your skills before proposing the next range of improvments at work.

Nothing to do with the coloured LED lights and racing stripes. Whoops, sorry wrong group of people.

Just when we thought spin speeds and storage density were peaking, along came the SSD. Solid State. Has a ring to it doesn’t it. It was stuff of labs and scifi for so long. So was Dick Tracy’s wrist watch (check out the pebble kickstarter project).

None of this waiting around for 1 or 2 minutes for my MacBook Pro to boot up. No way. That’s for the average Joe. I cant afford to wait more that 20 seconds. I need to rush out the door at 5:30pm (its okay, its all right now cos Sheryl says so) so that shutdown or VM suspend needs to fly.

Then there is the equivalent of the guys standing around your car with the hood up, admiring the chrome, the turbo charger, injectors etc.

In the end, it always comes down to speed and grunt. I wonder if neandertals compared their clubs in a similar fashion.

So our other half’s might think at best its an indulgence, at worst a complete waste of money. However, the skills acquired do pay dividends. Those in this fields who don’t keep up their professional proficiency risk their careers.

If you are considering the switch or already have, then hopefully I can save you some time and provide some guidance on how to proceed with tooling up for SSIS (or general Windows) development on this platform.

My reasons for choosing the MacBook Pro and subsequently using it as a Windows / SSIS development platform are simple. I really like it. It’s quality equipment, performs well and reliably and despite a few annoying niggles, its very productive.

So, if you have decided to go down this route, or are considering the switch, then read on.

The iPhone and iPad are driving the enormous growth in Mac sales via the ‘halo’ effect. At the other end of this discussion, Microsoft is absolutely dominating the ETL space with SSIS. If you have arrived at this blog post through a route other than SSIS, ETL means Extract, Transform & Load and SQL Server Integration Services (SSIS) is Microsoft’s solution.

In a nutshell, ETL is all about moving, consolidating & transforming data to make it more useful and valuable.

I used Windows based PC’s to do my ETL consultancy work (COBOL/SQL) right up until Apple decided to use Intel chips in their hardware platforms. I had always wanted to have a Mac but couldnt justify have two separate platforms. Enter Bootcamp.

Bootcamp allows a Mac OS X user (Tiger 10.4 onwards) to boot their system into Windows. This was a big step forward. I started with a Core 2 Duo, white Macbook with 4gig of RAM. Six years later I am using a 15″ MacBook Pro with 16gig of RAM, a 256gig SSD boot disk with a 750gb 7200rpm Momentus Hybrid hard drive in the DVD drive bay. This is all driven by OS X Lion (10.7) and allows me to run mulitple Windows XP and Windows 7 virtual machines on VMWare Fusion 4.1.1.

I also connect it to an Apple 27“ Thunderbolt display in my home office and to a 24” monitor (via a VGA adaptor) at a client site. The Apple Cinema / Thunderbolt displays are excellent. The resolution and image quality is so good. The Dell monitor is also fine, but not as good quality. However, there is a hefty price difference.

I was using two 24“ Apple Cinema displays connected via a Matrox DualHead2Go graphics device. This was great for parking SSMS in one screen and Visual Studio / BIDS in the other, with a bit of room for email and file explorer near the bottom. However, with the release of the Thunderbolt 27” monitor and its built in Ethernet, Thunderbolt and USB ports, I have gone back to the single, larger monitor. I keep the MBP screen open at the client site, using the 24″ monitor, so I still have a parking area for Mail and file Explorer. The Matrox device works with the new Thunderbolt comms port, but it did bring with it a few issues reviving the external monitor from sleep conditions. Matrox are fairly good with releasing firmware and software updates. I also make heavy use of Spaces, enabling fast switching between Windows 7, Windows XP and OS X Lion.

So thats the platform I have evolved over the last few years.

I think this is superb SSIS development option. As a data architect, communicating ideas, stratagies and plans are as important as the design and build process. This dual OS platform gives me the best of both worlds.

Next installment – Equipment and software list

Some big players out there simply attract attention because of their high price tags (it must be good right?). Informatica, Business Objects, Cognos, SAP and plenty of others offer the tantalising promise of end user BI solutions.

Recently there appears to have been significant activity from newer outfits deploying very compelling BI solutions, interfacing into SQL Server. They don’t all attempt to be all singing and dancing, but target specific needs within the BI arena. Varigence is one such example.

Whilst Oracle has been sitting on a ticking bug, possibly set to rival Y2K, IBM are what they are and niche vendors like Tandem NonStop SQL evoke questions like “who”, Microsoft has continued to roll out significant product improvements that genuinely raise the bar in corporate database solutions.

During a course I attended nearly a year ago, Ralph Kimball mentioned his “big data” white paper he had been commissioned to do by Informatic, casually dropping a few BIG company names and quoting outrageously BIGGER data volumes. It began to get airplay a few weeks later. Since then every vendor (and their best mate) have been dropping the “big data” buzzword.

Microsoft’s response has been to promote Vertipaq from SSAS into the core database engine. This is one of the biggest upgrades to the basic RDBMS infrasturcture in a very long. Previous enhancements across the industry have been focused on faster disk hardware, comms channels, RAM increases, very clever caching, DBA focused performance and tuning, SSD’s, virtualisation etc. I am sure there are plenty of other techniques and other vendors solutions won’t have been static.

But, ColumnStore goes right to the core of processing mind blowing datasets that are now becoming commonplace. Big data handling frameworks like MapReduce & then Hadoop kick started the trend, but Microsoft has delivered it into the heart of BI. Clearly they have been cooking this up for some time. If you want to read up on it then get yourself the free ebook released by Microsoft last week.

Dave Campbell, a Microsoft Technical Fellow, dropped this article on us early last week about “The coming in-memory database tipping point”. It was absolutely fascinating reading, especially for someone who “grew up” on b-trees, block splits, file reorgs, file contention measurement etc, on a system with only 512meg of memory per each of 8 CPU’s (and some of that had to be allocated as disk cache). I still can’t believe we are prepping a BI server with 96gig of RAM.

From what David has indicated, we have a lot more developments in this area to look forward to over the coming decade. This is one snippet, lifted directly from his article, of whats in store

Microsoft is also investing in other in-memory database technologies which will ship as the technology and opportunities mature. As a taste of what’s to come, we’re working on an in-memory database solution in our lab and building our real-world scenarios to demonstrate the potential. One such scenario, based upon one of Microsoft’s online services businesses, contains a fact table of 100 billion rows. In this scenario we can perform three calculations per fact – 300 billion calculations in total, with a query response time of 1/3 of a second. There are no user defined aggregations in this implementation; we actually scan over the compressed column store in real time.

As John would say, “Thats awesome dude”.