I tweeted yesterday about how i wished I had known about pairwise testing earlier, so perhaps a little explanation would help.
I attended at Scottish Test Forum event yesterday in Glasgow.
One of the presentations was given by Stephen Allot from Electromind and covered pairwise testing. Presentations will be available at http://www.bcs.org/server.php?show=nav.9979 in near future.
I hadn't come across it before and basically is a technique that assumes that almost all defects are due to one error or a combination of two errors and so we can apply the mathematics of pairs to reduced the number of tests required.
So if you are trying to test a situation with a number of parameters each with different possible values then the total possible combinations multiply up to huge numbers which become impossible or at least impractical testing, but pairwise can reduce that to much smaller and achievable numbers of tests needed.
The mathematics of pairs is well known (well its well known to mathematicians ;-) ) and goes back to Latin squares and orthogonal arrays and all that.
Once you've decided to use pairwise, there are lots of tools available (easy to find try Wikipedia or http://pairwise.org/ or http://www.satisfice.com/ etc.) that can take your parameters and produce the lists of tests. Some are freeware and some cost, the benefits of paying being that you can get better lists (shorter and with illegal combinations excluded).
An example of this is that if you had 10 parameters each with 10 values then the free tools would give you 177 tests to execute, whereas the paid-for tools could reduce that to 140. However if you compare that to the total combinations of 10 billion, then the difference between free and not is minor.
So at this point its a wow, what great reductions, but where did that assumption come from?
We are that assuming that almost all defects are due to one error or a combination of two errors. the answer I got was that it was all based on engineering and medical experience. Wikipedia showed the way to an IEEE paper that brought the research together
http://csrc.nist.gov/groups/SNS/acts/documents/TSE-0172-1003-1.pdf
The conclusion was that testing single errors only covered 68%, covering for pairs of errors covered 93%, triples covered 98% and quadruples 100%.
So a combination of pairwise testing plus defined test cases for known corner point issues should get pretty close to complete coverage with a fraction of the testing.
However the basis is all empirical so you will always have people saying its not applicable, and even the IEEE paper ends with "many more empirical studies of other classes of software are needed to evaluate the applicability of combinatorial testing for other classes of systems" so you must use the technique with your brain switched on.
Wednesday, 25 November 2009
Monday, 23 November 2009
Thoughts on the future of Twitter
Three separate tweets arrived today. And I felt they could be combined to paint wildly different possible views of the future of Twitter.
First of all, came the thought that Twitter would be sold next year for up to 2 billion dollars (@Twitter_Tips Microsoft or Google Will Buy Twitter in 2010 http://j.mp/5nKRij). I find that a bit hard to agree with as without hard data on the usage and a real monetization plan, then what would they be buying. On the user count front almost every day you will see comments about the number of users on Twitter rising or falling, but there are no hard facts behind the discussions so everyone brings their own baggage and jumps onto their hobby horse. The variations of usage of web vs api vs whatever means that the numbers that are available need to have so many assumptions and caveats applied then you could argue anything with them, but the shift in demographics from older to younger users does seem to suggest a change in usage at the very least).
The second tweet suggested that Twitter is a giant pyramid scheme with risk of collapsing (@Twitter_Tips If We All Quit Listening & Only Broadcast, Is Twitter A Giant Ponzi scheme? http://j.mp/5maN4U) as users don't post and get fed up of the marketing being published at them. Again I don't totaly buy it, however as a relative newbie I have been shocked at how few people actually tweet and how their geographical distribution is uneven to say the least, so I don't think there is a firm foundation.
But surely Twitter couldn't have any real problems.
Look at how many users and traffic it has.
Which brings me to today's third tweet about the problems of the former darling of social networking MySpace (@bcs MySpace must 'regain its mojo': 23/11/2009 http://bit.ly/08B6cei).
so where does that leave us?
Twitter could have a bright future with funding from a giant, or it could disappear perhaps in the undercurrent from Google Wave. The concept of microblogging is great, however most of the tweets don't stick to the 140 characters, they refer you to a "proper" blog with the complete story. So if Twitter isn't for telling people what you are doing and is just for pointing you to other information, then what is its killer app reason to exist?
And at this point I have to say I don't read Hello and if the killer app is just to find out what the celebs had for breakfast then maybe its a club that I wouldn't want to be part of ;-)
But in the meantime I'm getting real value from the tool and pleasure from the wide range of tweets I see and have managed to miss most of the mass-marketing and DMs that other users complain about.
First of all, came the thought that Twitter would be sold next year for up to 2 billion dollars (@Twitter_Tips Microsoft or Google Will Buy Twitter in 2010 http://j.mp/5nKRij). I find that a bit hard to agree with as without hard data on the usage and a real monetization plan, then what would they be buying. On the user count front almost every day you will see comments about the number of users on Twitter rising or falling, but there are no hard facts behind the discussions so everyone brings their own baggage and jumps onto their hobby horse. The variations of usage of web vs api vs whatever means that the numbers that are available need to have so many assumptions and caveats applied then you could argue anything with them, but the shift in demographics from older to younger users does seem to suggest a change in usage at the very least).
The second tweet suggested that Twitter is a giant pyramid scheme with risk of collapsing (@Twitter_Tips If We All Quit Listening & Only Broadcast, Is Twitter A Giant Ponzi scheme? http://j.mp/5maN4U) as users don't post and get fed up of the marketing being published at them. Again I don't totaly buy it, however as a relative newbie I have been shocked at how few people actually tweet and how their geographical distribution is uneven to say the least, so I don't think there is a firm foundation.
But surely Twitter couldn't have any real problems.
Look at how many users and traffic it has.
Which brings me to today's third tweet about the problems of the former darling of social networking MySpace (@bcs MySpace must 'regain its mojo': 23/11/2009 http://bit.ly/08B6cei).
so where does that leave us?
Twitter could have a bright future with funding from a giant, or it could disappear perhaps in the undercurrent from Google Wave. The concept of microblogging is great, however most of the tweets don't stick to the 140 characters, they refer you to a "proper" blog with the complete story. So if Twitter isn't for telling people what you are doing and is just for pointing you to other information, then what is its killer app reason to exist?
And at this point I have to say I don't read Hello and if the killer app is just to find out what the celebs had for breakfast then maybe its a club that I wouldn't want to be part of ;-)
But in the meantime I'm getting real value from the tool and pleasure from the wide range of tweets I see and have managed to miss most of the mass-marketing and DMs that other users complain about.
Saturday, 21 November 2009
Free Conferences
Earlier today I was reading the Financial Times (unusual for me, but I found that the Saturday paper is a good read).
One article that caught my eye was Mike Southon's Take the free and easy route
http://www.ft.com/cms/s/0/130b6ddc-d5fa-11de-b80f-00144feabdc0.html
where he looks at the concept of “free” business models especially in relational to conferences.
His conclusion was that the approaches to use were things like that
- specific free offers to drive business for his members:
- congress centres could offer unused space to the local community;
- or hotels and venues could waive charges on marginal items, such as wi-fi.
The International Congress and Convention Assoviation (ICCA) uses free models to good effect, offering one free congress registration to all first-timers in the year they join, an extensive PR kit with details of more than 350 media organisations and an online destination comparison system.
Which led to a 20% increase in attendees.
But I felt that Mike missed the point. The free business model concept of "offer(ing) everything for free first and build a business model later – as Google, Facebook and YouTube have done" takes you to a different place.
The real free conference approach would be focused on removing conference fees and so increasing attendance. As the attendee volumes should be higher, the charges for commercial exhibition space could be higher which could cover the fixed conference charges.
Obviously the topic has to be something of interest.
But I've seen this work on things like the cloud computing conference Cloud Scotland, OK on a small scale but it could work on a bigger scale as well.
Its a different sort of conference and must be considered valuable to be successful, so there is a real balancing game to play.
One article that caught my eye was Mike Southon's Take the free and easy route
http://www.ft.com/cms/s/0/130b6ddc-d5fa-11de-b80f-00144feabdc0.html
where he looks at the concept of “free” business models especially in relational to conferences.
His conclusion was that the approaches to use were things like that
- specific free offers to drive business for his members:
- congress centres could offer unused space to the local community;
- or hotels and venues could waive charges on marginal items, such as wi-fi.
The International Congress and Convention Assoviation (ICCA) uses free models to good effect, offering one free congress registration to all first-timers in the year they join, an extensive PR kit with details of more than 350 media organisations and an online destination comparison system.
Which led to a 20% increase in attendees.
But I felt that Mike missed the point. The free business model concept of "offer(ing) everything for free first and build a business model later – as Google, Facebook and YouTube have done" takes you to a different place.
The real free conference approach would be focused on removing conference fees and so increasing attendance. As the attendee volumes should be higher, the charges for commercial exhibition space could be higher which could cover the fixed conference charges.
Obviously the topic has to be something of interest.
But I've seen this work on things like the cloud computing conference Cloud Scotland, OK on a small scale but it could work on a bigger scale as well.
Its a different sort of conference and must be considered valuable to be successful, so there is a real balancing game to play.
Friday, 20 November 2009
How do you find free events?
Well, its a follow up question to the last one.
how did you find out about the seminars in the first place (i.e. where did you look or hear about them in order to sign up to attend)?
So I guess its all part of the networking stuff.
I am a member of the BCS and attend its branch meetings.
I'm also a member of a reasonably number of LinkedIn Groups and connected to over 200 people on LinkedIn.
And an alumni of Glasgow and Strathclyde universities.
Because of those contacts I get a continual stream of emails coming through.
The local ones I check carefully for what's on, the Linked-In updates will also show events that contacts are planning to attend.
And then I have a diary, when I hear about an event I'll look it up on the web and if remotely interesting write it down in the diary. As we get closer to the date I'll check if there's anything else conflicting and book in.
Another way to find events is when you find a new organization (for instance Scottish Developers) look up their events list, look up their links to other organizations. And if they are using a local booking tool (like eventbrite or something like that) then see what other events are available.
There are events on LinkedIn, but I've not found that too successful.
And once you've found an interesting source, keep checking it.
John
how did you find out about the seminars in the first place (i.e. where did you look or hear about them in order to sign up to attend)?
So I guess its all part of the networking stuff.
I am a member of the BCS and attend its branch meetings.
I'm also a member of a reasonably number of LinkedIn Groups and connected to over 200 people on LinkedIn.
And an alumni of Glasgow and Strathclyde universities.
Because of those contacts I get a continual stream of emails coming through.
The local ones I check carefully for what's on, the Linked-In updates will also show events that contacts are planning to attend.
And then I have a diary, when I hear about an event I'll look it up on the web and if remotely interesting write it down in the diary. As we get closer to the date I'll check if there's anything else conflicting and book in.
Another way to find events is when you find a new organization (for instance Scottish Developers) look up their events list, look up their links to other organizations. And if they are using a local booking tool (like eventbrite or something like that) then see what other events are available.
There are events on LinkedIn, but I've not found that too successful.
And once you've found an interesting source, keep checking it.
John
Are free seminars worth it?
Anyone that has been following my tweets or linked-in status will have noticed that I've upped my attendance at seminars, briefings, exhibitions, etc. recently.
I was asked by one of my friends how I am finding these seminars and instead of replying directly I thought I'd blog about it (as a new blogger I haven't really got the knack yet of jumping into text to schedule, I need an idea to get me started).
The short answer is that they are well worthwhile, there isn't a single one of the recent sessions I've been to that I didn't get something out of.
But as they say, "you don't get nowt for nowt", you have to aware of the motives of the organisers.
For instance I went to a briefing from Gartner and half of it described what was going on in the area and the other half was a "and here's how we can help you". I didn't mind the sales pitch as it wasn't that overt.
Associations and alumni events are usually neutral (I've attended ones by AMBA, BCS and Strathclude University Business School), although perhaps not as pointed and focused and leading edge as they could be as they are aimed at a more generalist audience.
Government sponsored events are good to see where there is focus and so likely to be public money and consequently business opportunities, it Britain there is a Knowledge Transfer program. I went to a Digital Communications KT event, very interesting, although it almost seemed like a clique attends them.
User groups are also good. I'm a member of the Scottish Test Group and their sessions cover the width of the activity so there's always been stuff of interest.
But if you have an area of interest you can find sessions and gain a huge amount of info. I wanted to pick up a bit more info on Agile, from the BCS I learnt about the Agile Way of Thinking, from Scottish Test Group I learnt about Test Driven Development (TDD), from Agile Scotland I got a more techy view of TDD and over the next week or two I'll get an overview of Agile from the Rational User Group (I'm not a member but its open and I expect a bit of sales pitch from IBM but that's OK) and a much more techy set of info about TDD from Scottish Developers.
If you are willing to put in the time you can get info and widen your knowledge on what's going on.
And often its the audience that provides the best comments. The Test Manager at Dell in Glasgow made the comment that it was the next generation of TDD based on requirements that she was after and it wasn't there yet and as that lines up exactly with what I was looking for, it shows me that there is a real desire to take TDD further and so this is an are I will keep an eye on.
I was asked by one of my friends how I am finding these seminars and instead of replying directly I thought I'd blog about it (as a new blogger I haven't really got the knack yet of jumping into text to schedule, I need an idea to get me started).
The short answer is that they are well worthwhile, there isn't a single one of the recent sessions I've been to that I didn't get something out of.
But as they say, "you don't get nowt for nowt", you have to aware of the motives of the organisers.
For instance I went to a briefing from Gartner and half of it described what was going on in the area and the other half was a "and here's how we can help you". I didn't mind the sales pitch as it wasn't that overt.
Associations and alumni events are usually neutral (I've attended ones by AMBA, BCS and Strathclude University Business School), although perhaps not as pointed and focused and leading edge as they could be as they are aimed at a more generalist audience.
Government sponsored events are good to see where there is focus and so likely to be public money and consequently business opportunities, it Britain there is a Knowledge Transfer program. I went to a Digital Communications KT event, very interesting, although it almost seemed like a clique attends them.
User groups are also good. I'm a member of the Scottish Test Group and their sessions cover the width of the activity so there's always been stuff of interest.
But if you have an area of interest you can find sessions and gain a huge amount of info. I wanted to pick up a bit more info on Agile, from the BCS I learnt about the Agile Way of Thinking, from Scottish Test Group I learnt about Test Driven Development (TDD), from Agile Scotland I got a more techy view of TDD and over the next week or two I'll get an overview of Agile from the Rational User Group (I'm not a member but its open and I expect a bit of sales pitch from IBM but that's OK) and a much more techy set of info about TDD from Scottish Developers.
If you are willing to put in the time you can get info and widen your knowledge on what's going on.
And often its the audience that provides the best comments. The Test Manager at Dell in Glasgow made the comment that it was the next generation of TDD based on requirements that she was after and it wasn't there yet and as that lines up exactly with what I was looking for, it shows me that there is a real desire to take TDD further and so this is an are I will keep an eye on.
BI The Big Discrepancy
(originally posted at http://bit.ly/434VV7)
I attended a Gartner briefing on the 19th of November on the topic of Business Intelligence that promised their take on why many organizations still feel data rich and information poor.
The briefing took the form of two presentations.
The first was from James Richardson on the subject "Too Much Technology and Not Enough Intelligence".
James is an experienced BI man and has been in Gartner a couple of years.
His point was that BI has been #1 on the CIO list of priorities for many years and many companies have worked on it, but it hasn't delivered the expected results.
One repeated thought was that BI can't be an IT project, it needs to have heavy business involvement and ownership.
They also made the point that it shouldn't be a tools question. The technology is mature and the suppliers are established (7 of 13 main suppliers have been assessed as market leaders level, meaning that although they each have advantages there isn't a lot to differentiate them).
Organizations should have a BI strategy and then align the tool suppliers with that strategy, rather than choosing a supplier and doing whatever they are best at. Gartner, of course ;-), can help with this and there is a framework document on Gartner.com that will help you define the strategy.
There are many reasons for doing BI (reduce costs, increase speed, improve quality, etc). While these are all admirable the point that Gartner made was that they are to some extent mutually exclusive and you should decide on why you are doing BI and if you focus on that then that is how you will get success, trying to be all things to all people can be a recipe for disaster.
They feel that having a BI competency centre is critical to success. This is a joint biz/IT team that handles all aspects of the BI.
Fatal flaws in implementing BI
- if you build it they will come
- managers needing to "dance with the numbers" (the info that key spreadsheet jockeys have that creates individual departmental view of the data should be co-opted into the competency centre and included in the BI solution).
- data quality problem? What data quality problem? (need active data stewards who can articulate and communicate the problems and assumptions and meta-data about their data)
- Evaluate other BI platforms? Why bother?
- its perfect as it is - don't ever change
- let's just outsource the whole darn BI thing (but its a core competency so that's a bad idea)
- Just give me a dashboard - NOW!
- Err X + Y = Z, doesn't it (enterprise metrics framework based around biz values not technology)
- BI strategy - no thanks, we'll just follow our nose.
I thought that the dashboard approach may be a way to surface the differences in departmental views and definitions and also identify the key underlying technology issues, so i disagreed to a certain extent with that point, however the issue that was being made was that a cobbled-together dashboard could be retained forever rather than just being a short-duration trial and that may suck resources into supporting an inefficient and impractical solution.
The second half was given by Simon Greener and this was a view of a number of case studies.
In many ways this was much more the pitch for what Gartner can do and what services it can offer to help you on BI.
I didn't note as much from that presentation
- developing BI agenda needs clear strategy.
- conduct a BI maturity assessment (grades where you are on a scope of 1-5 on 3 dimensions)
- Heading for world class (level 5 on all 3 dimensions) may not be the best initial goal.
- BI strategy must be one that engages with the business - not by IT for IT about IT.
Corporate Performance Management systems were also mentioned as one of the solutions in certain circumstances.
The session ended with a pitch for the Gartner Business Intelligence Summit 2010.
all in all a good overview, maybe nothing earth shattering but interesting to hear the difference between IT and Business priorities mentioned again and again and really that's the key reason for the Big Discrepancy.
John Bryden
jb37432@yahoo.com
I attended a Gartner briefing on the 19th of November on the topic of Business Intelligence that promised their take on why many organizations still feel data rich and information poor.
The briefing took the form of two presentations.
The first was from James Richardson on the subject "Too Much Technology and Not Enough Intelligence".
James is an experienced BI man and has been in Gartner a couple of years.
His point was that BI has been #1 on the CIO list of priorities for many years and many companies have worked on it, but it hasn't delivered the expected results.
One repeated thought was that BI can't be an IT project, it needs to have heavy business involvement and ownership.
They also made the point that it shouldn't be a tools question. The technology is mature and the suppliers are established (7 of 13 main suppliers have been assessed as market leaders level, meaning that although they each have advantages there isn't a lot to differentiate them).
Organizations should have a BI strategy and then align the tool suppliers with that strategy, rather than choosing a supplier and doing whatever they are best at. Gartner, of course ;-), can help with this and there is a framework document on Gartner.com that will help you define the strategy.
There are many reasons for doing BI (reduce costs, increase speed, improve quality, etc). While these are all admirable the point that Gartner made was that they are to some extent mutually exclusive and you should decide on why you are doing BI and if you focus on that then that is how you will get success, trying to be all things to all people can be a recipe for disaster.
They feel that having a BI competency centre is critical to success. This is a joint biz/IT team that handles all aspects of the BI.
Fatal flaws in implementing BI
- if you build it they will come
- managers needing to "dance with the numbers" (the info that key spreadsheet jockeys have that creates individual departmental view of the data should be co-opted into the competency centre and included in the BI solution).
- data quality problem? What data quality problem? (need active data stewards who can articulate and communicate the problems and assumptions and meta-data about their data)
- Evaluate other BI platforms? Why bother?
- its perfect as it is - don't ever change
- let's just outsource the whole darn BI thing (but its a core competency so that's a bad idea)
- Just give me a dashboard - NOW!
- Err X + Y = Z, doesn't it (enterprise metrics framework based around biz values not technology)
- BI strategy - no thanks, we'll just follow our nose.
I thought that the dashboard approach may be a way to surface the differences in departmental views and definitions and also identify the key underlying technology issues, so i disagreed to a certain extent with that point, however the issue that was being made was that a cobbled-together dashboard could be retained forever rather than just being a short-duration trial and that may suck resources into supporting an inefficient and impractical solution.
The second half was given by Simon Greener and this was a view of a number of case studies.
In many ways this was much more the pitch for what Gartner can do and what services it can offer to help you on BI.
I didn't note as much from that presentation
- developing BI agenda needs clear strategy.
- conduct a BI maturity assessment (grades where you are on a scope of 1-5 on 3 dimensions)
- Heading for world class (level 5 on all 3 dimensions) may not be the best initial goal.
- BI strategy must be one that engages with the business - not by IT for IT about IT.
Corporate Performance Management systems were also mentioned as one of the solutions in certain circumstances.
The session ended with a pitch for the Gartner Business Intelligence Summit 2010.
all in all a good overview, maybe nothing earth shattering but interesting to hear the difference between IT and Business priorities mentioned again and again and really that's the key reason for the Big Discrepancy.
John Bryden
jb37432@yahoo.com
Subscribe to:
Posts (Atom)