Refining the group test format

Comments Off

Posted on 27th December 2008 by Simon Joinson in Camera reviews |Syndicated Press

It's long been obvious that the existing dpr review format - even in its seriously truncated 'concise' form - is unsuitable for the majority of what can be broadly termed 'point and shoot' compact cameras, and I've been wanting to develop a group test format for some time now. There are simply too many compact cameras released each year for us to keep up with (even the most basic camera takes a minimum of a week to put through our battery of tests and to produce a gallery), and the truth is that image quality amongst competing models doesn't - for the most part - vary that much. But there are always going to be exceptions, cameras with image quality that falls well below or rises way above the mean, and unless we test a far higher proportion of the models on the market there's a risk we'll miss these occasional stars and lemons. The group test format may not give you as much detail about the individual cameras, but it does give you a much better idea of the relative quality of a far greater number of competing models.
The recently published group tests were designed specifically to offer a snapshot of the market in the run up to the peak buying season, and to help the huge number of dpreview visitors who arrive looking for help choosing a camera - those that aren't regulars or forum activists. The format was chosen to reflect the way digital cameras are typically used, and to compare how the various models performed in everyday situations. Unusually for dpreview the image quality was assessed first and foremost in terms of typical viewing magnifications, rather than at a pixel level, with the emphasis on accuracy and reliability of the autofocus, metering and white balance systems and on issues that impact on the quality of a typical print (color, contrast, dynamic range and so on). This doesn't reflect a shift in the dpreview ethos or any intention to change the way our full reviews are conducted; it simply recognizes that the typical buyer of a budget digital camera - or one designed to slip into a shirt pocket - is less likely to be concerned with pixel peeping than the typical dpreview regular.
We've grown to expect anything and everything we publish to receive far more brickbats than bouquets on the forums - a thick skin is an essential attribute for anyone writing for dpreview - but our impression is that the idea of a group test has been fairly well received (and they've certainly had a lot of traffic), even if there are disagreements on the cameras chosen and the conclusions drawn. We're almost certainly going to be doing more of them in the future, and to be honest I can't see many simple point and shoot cameras getting full reviews given the number of DLSR and 'prosumer' compacts launched each year.
So now we're working on refining the group test format to offer a little more than those just published without getting to the point where the articles become unwieldy. I've actually delayed publication of the last group test (the super zooms) so we can add a couple more tests, as befits cameras that in many cases are almost as sophisticated as the typical entry-level DSLR.
I'm sure everyone has their own idea of what's important and of what should be included in the group tests, and we have been following the feedback on the current crop closely. For the most basic cameras I think the format used for the articles just published is probably sufficient, but for more advanced cameras we probably need to add at least some of the following to what we included this time around:
  • Raw comparison
  • Noise analysis
  • Basic performance measurements and timings
  • Optical aberrations (CA etc)
  • Distortion
  • Dynamic range
  • Macro performance
  • Resolution
  • Movie mode
This is what we're currently debating, and hopefully you'll see the group test format developing into the genuinely useful comparison tool we envisage. We also see the group test format as being something we could possibly add to our full SLR reviews, allowing us to, for example, gather together all the entry-level digital SLRs we've already reviewed to produce a more concise comparison article once or twice a year.
    Obviously high-end compacts such as the Canon G10 and Panasonic LX3 will still warrant full reviews, as will many of the superzoom cameras, and the more advanced premium models, but for the scores of other compacts launched each year the group test format will allow us to broaden our coverage to a significantly bigger proportion of the market without compromising our 'get under the skin' approach to reviewing.

    Ten years on

    Comments Off

    Posted on 25th December 2008 by Phil Askey in Camera reviews |Syndicated Press

    Ten years, nobody who starts a web site can honestly claim to have an accurate vision of what that site will be like ten years down the line.  When I wrote my first digital camera article and dropped it onto the now defunct photo.askey.net URL I had no intentions for it to be anything more than an opinion piece of my purchase. But then a funny thing happened, I began getting emails, lots of them, all asking about my article and what I thought of other cameras.  It seemed that this "digital camera thing" was becoming popular and what people were looking for was a site which was always up to date and could deliver honest, detailed reviews.  I went out on a limb and spoke to several manufactures, to my surprise the majority were extremely receptive and before I knew what was happening I had a steady flow of cameras coming my way.  Over the next couple of years I developed the primary site features (camera database, forums, news, etc.) as well as gradually improving the testing methodology (but at all times trying to be consistent).

    The hard work began to pay off, we saw our visitor figures skyrocket; 3 million page views per month after a year, 11 million after two years, today we serve around 140 million pages per month to over 7 million unique visitors (or about 24 million visitor sessions if you prefer).  In 2004 having done virtually a hundred hours a week for five years I decided getting some help may be a good idea and was extremely lucky to hire someone with a long standing digital camera review pedigree; Simon Joinson.  In 2006 I was approached by Amazon to discuss potential acquisition, for me, and for the site, this was actually perfect timing, we were massively resource limited, needing a new office, more staff, more development and new infrastructure.  In the eighteen months since the deal was signed the way we work has changed completely, we now have a large dedicated office and studio, totally new server infrastructure (hosted in the same facility as Amazon) as well as four camera reviews, a lens reviewer, a studio assistant / news author and two (soon to be three) developers.

    Naturally I'm very proud of how far we've come, but I am also aware (especially in tough times such as these) that the need to constantly innovate and change never goes away, proof of this can be seen in our recent group test series which is a totally new format to dpreview but has been delivered to perfection by Simon and the team.  You should also expect to see a lot of new development next year, not least of which our new Challenges system which we'll be launching into public beta on 29th December.

    What’s in a (lens) review?

    Comments Off

    Posted on 3rd December 2008 by Andy Westlake in Lens reviews |Syndicated Press

    Hot on the heels of publishing a new lens review (the Tamron 18-270mm F3.5-6.3 VC), I thought our readers might be interested in getting an insight into the overall review process and the amount of work involved. While some of our more naive forum posters appear to labour under the misapprehension that our reviews are just a few days work conducted entirely in the studio, the reality is that the process is a lot more complicated than that. In fact the experience gained from going out and actually using the product is also very important, and informs the final conclusion just as much as all of that studio work. So in a nutshell, here is what the review process actually involves.

    Step One - Product shots

    The first step when a lens arrives is to take all of the product shots which are used on the front page and the 'Body and Design' page. This is best done when the lens is still in pristine cosmetic condition, as it can sometimes get marked during the course of testing. While there's only about 10-15 different views used per review, each is shot multiple times. Total number of shots for this review was 45 (obviously of the lens, not with it).

    Step Two - Studio Tests

    Producing the profile data that gets displayed in the lens review widget is a time-consuming process. Test charts have to be very precisely aligned and focused, and the data shot at least twice to assure reproducibility.

    Resolution and CA - 7 marked focal lengths, at least 7 aperture settings at all focal lengths (and 8 at 50mm). Two complete data sets shot = 100 shots precisely
    Distortion - This requires the most critical alignment to make the data as useful as possible. 7 marked focal lengths, but an average of 14 shots at each to achieve optimum alignment. As it happens, also 100 shots total.
    Falloff - 7 marked focal lengths, third-stop increments from wide open to F11, data obtained from averaging three exposures. This required a total of 171 shots.
    Macro - the 18-270mm, with its rather extreme focus shift on stopping down, was a tricky lens for this deceptively simple test. Eventually, to work out precisely what was going on, we resorted to shooting the macro chart using a focusing rail with the lens set to its minimum focus distance. Shots were taken at each full stop in the aperture range, at 1mm distance increments. This procedure generated 77 shots to define the lens's behavior accurately. A further set of 14 shots of a 3-dimensional test subject were also taken to illustrate the focus shift visually. Total = 91 shots.
    Image stabilization - 10 shots at each of five shutter speeds with IS on and off, repeated for three focal lengths. 10x5x2x3 = 300 shots

    Together, this gives a grand total number of shots used in the studio tests of 762.

    Step Three - Real World Shooting

    One of the more mystifying criticisms we hear of our reviews is that they aren't from 'real users'. Strangely enough each of the reviewers here is (probably) a 'real' person, and we tend to carry the camera/lens we're working on around with us pretty well all of the time. It's currently late autumn (fall for our transatlantic friends)/early winter here in London, which means the weather is mainly gray and damp, with the occasional bright but cold day thrown in. Good shooting conditions are therefore rare, and we have to make use of them whenever they turn up.

    Even so, I managed to fit in several good sessions with the Tamron 18-270mm. I actually prefer to shoot with the lens fairly extensively before processing all of the studio data, to get an initial impression of how it performs which isn't colored by those results. Once I've digested all of the test data I'll then go out looking to illustrate any specific weaknesses identified in the tests (and if I don’t find them, repeat the tests). Generally the studio data is remarkably predictive of how the lens will perform in actual use, but this ‘shoot, test, shoot, test’ approach mitigates well against drawing incorrect conclusions based on just the studio data alone. It’s also important to realize that a full lens review simply can’t be done purely in a studio environment: the technical data is hugely valuable but it says nothing about autofocus speed under various lighting conditions, or flare, or bokeh; for these we need to go out and take 'real' pictures.

    In the case of the Tamron 18-270mm, I shot a total of about 500 'real-world' exposures, 200 on the D90 and 300 on the D300. Normally I would also aim to shoot on a lower end body too (i.e. the D60), but didn't get a chance in the timescale of this review.

    So in summary, that's the work that goes into delivering you a lens review. In this example it adds up to about 1250 shots with the lens under test, in the course of slightly less than three weeks. I suspect this is rather more intensive use and analysis than most 'real users' ever manage, and it’s the sum total of all of this experience that goes into writing the final review.

    African Photography Blog – South Africa Photographic News, Articles, Tutorials is Digg proof thanks to caching by WP Super Cache