Google PlusFacebookTwitter

Amazon’s Fire Phone is incredibly smart…and what it means for the future of smartphones

By on Aug 25, 2014 in Tech Takes | 2 comments

The announcement of the Amazon Fire Phone is one of the most interesting technology news I’ve come across in recent times. While the jury is out on whether it will be a commercial success or not (UPDATE: recent estimates suggest it could be as low as 35,000), the features that the phone comes with got me thinking about the technical advancements that have made it possible. The caveat here is much of what follows is speculation – but I do have a background in research projects in speech recognition and computer vision related user experience research. I’m going to dive into why Fire Phone’s features are an exciting advance in computing, what it means for the future of phones in terms of end-user experience, and a killer feature I think many other pundits are missing out. Fire Phone’s 3D User Interface I did my final year research project on using eye tracking on mobile user interfaces as a method of user research. The problem with many current methods of eye tracking is that it requires specialised hardware – typically the approach is to use a camera that can “see” in infrared, illuminate the user’s eye using infrared, and using the glint from the eye to track the position of the eyeball relative to the infrared light sources. This works fabulously when the system is desktop-based. Chances are, the user is going to be within a certain range of distance from the screen, and facing it at a right angle. Since the infrared light sources are typically attached to corners of the screen – or an otherwise-known fixed distance – it’s relatively trivial to figure out the angles at which a glint is being picked up. Indeed, if you dive into research into this particular challenge in computer vision, you’ll mostly find variations of approaches on how to best use cameras in conjunction with infrared. The drawback to this approach is that the complexity involved vastly increases when it comes to mobile platforms. To figure out the angle at which glint is being received, it’s necessary to figure out the orientation of the phone from it’s gyroscope (current position) and accelerometer (how quickly the pose of the phone is changing in the world). In addition to this, the user themselves might be facing the phone at an angle rather than facing it at a right angle, which adds another level of complexity in estimating pose. (The reason this is needed is to estimate visual angles.) My research project’s approach was using techniques similar to a desktop-based eye tracking software called Opengazer coupled with pose estimation in mobiles to track eye gaze. Actually, before the Amazon Fire Phone there’s another phone which touted it had “eye tracking” (according to the NYT): Samsung Galaxy S IV. I don’t actually have an Samsung Galaxy to play with – nor did the patent mentioned in the New York Times article link above show any valid results – so I’m basing my guesses on demo videos. Using current computer vision software, given the proper lighting conditions, it’s easy to figure out whether the “pose” of a user’s head has changed: instead of a big, clean circular eyeball, you can figure out there’s an oblong eyeball instead which suggests the user has tilted their head up or down. (The “tilt device” option for Samsung’s Eye Scroll, on the other hand, isn’t eye tracking at all as it’s just using the accelerometer / gyroscope to figure out the device is being tilted.) What I don’t think the Samsung Galaxy S IV can do with any accuracy is pinpoint where a user is looking at the screen beyond the “it’s changed from a face at right angle to something else”. What makes the Fire Phone’s “3D capabilities” impressive? Watch the demo video above of Jeff Bezos showing off the Fire Phone’s 3D capabilities. As you can see, it goes beyond the current state-of-the-art that the Galaxy S IV has – in the sense that to accurately follow and tilt the perspective based on a user’s gaze, the eye tracking has to be incredibly accurate. Specifically, instead of merely basing motion on how the device is tilted or how the user moves their head from a right angle perspective, it needs to combine device tilt pose, head tilt / pose, as well as computer vision pattern recognition to figure out the visual angles the user is looking at an object from. Here’s where Amazon has another trick up its sleeve. Remember how I mentioned that glints off infrared light sources can be used to track eye position? Turns out that the Fire Phone uses precisely that setup – it has four front cameras, each with its own individual infrared light source to accurately estimate pose along all three axes. (And in terms of previous research, most desktop-based eye tracking systems that are considered accurate also use at least three fixed infrared light sources.) So to recap, here’s my best guess on how Amazon is doing it’s 3D shebang: Four individual cameras, each with it’s own infrared light source. Four individual image streams that need to be combined to form a 3D perspective… …combined with device position in the real world, based on its gyroscope… …and how quickly that world is changing based on its accelerometer Just dealing with one image stream alone, on a mobile device, is a computationally...

Exun 2013

By on Dec 4, 2013 in Tech Takes | 3 comments

This weekend, I was at Exun 2013, one of Delhi’s biggest computer technology symposiums (along with Code Wars). Having been a participant at the event for many, many years it felt nice to be back as a judge at Exun and meet so many bright kids into technology. I signed on for Exun when DPS RK Puram’s HoD, Mr Mukesh Kumar, got in touch with me a couple of weeks ago about conducting the junior quiz, senior quiz, and crossword. I expected nothing short of the best teams at this event, hence why I knew I needed to put in extra effort to ensure the tradition of Exun’s event standards were maintained. It was an amazing – and tiring – experience to conduct the three events, but I loved every minute of it. Raghav Khullar helped me build the question archive for all events, and Exun members / Mukesh sir helped me with organisational logistics at every stage. And with that, I present the archives for Exun 2013: Junior Quiz: Prelims (PDF, ~370 KB); Finals (PDF, ~3.3 MB) Crossword: Prelims (opens in a new window); Finals (opens in a new window) n.b. I’m aware of an error in the one question in the finals, where the answer should have been “SILKROAD” and not “SILKROUTE” Senior Quiz: Prelims (PDF, ~680 KB); Finals (ZIP, ~6.7 MB) n.b. I had to use PPTX for the finals presentation decks because it contains embedded media. Hope the teams enjoyed the quizzes and the crossword. Feedback appreciated! :) This slideshow requires...

Lessons in smartphone videography: What I learnt from editing a video shot on a phone

By on Aug 10, 2013 in Tech Takes | 2 comments

I have been obsessed for a while now with the idea of making a short film shot entirely on a smartphone. The versatility that a phone would allow in sheer ease of organising filming schedules is what attracts me the most. There has been a significant amount of interest from amateur / professional filmmakers in the industry along similar lines. What I did not want to do, however, is to plan a shoot, film it on a smartphone, and then end up with a substandard product. I needed a low-risk project to try out my idea on. Fortunately, the chance presented itself when I got the idea of recording my time at the university’s Graduation Ball: I would record videos of my friends talking about the first time they met me, and what they thought of me. The beauty of this plan was that due to filming times, I would get to put my smartphone camera through its paces in a multitude of lighting conditions and noise environments, and since the video was unscripted, the content of the videos need not be a “good” or a “bad” take. You can watch The First Time…At Grad Ball to see what my effort worked out as. (The second part of my The First Time project was a photo album on Facebook telling the story of the first time I met my friends and what I thought of them.) I realise that the video itself is pretty much a vanity project. Yet, I felt genuinely happy to do this because for the first time in my life – because of psychological issues I’ve had – I actually feel connected to my friends; that I care about them as individual beings. I wanted to create something to capture the essence of those emotions that I felt. Graduation, even though I’m not graduating yet, is one time people are allowed to be sentimental. But I digress. The point of this blog post is to document my experience of what I learnt through the process of filming and editing video on a smartphone. The “Why Now” On Smartphone Filming Mobile phones have been able to record videos for around six years now, so it’s interesting to note how in general there’s a lot more buzz now about using them in amateur / prosumer contexts. Part of this comes from the fact that while early “smart” mobile phones (think Symbian and their ilk) could record video, the de facto recording format was 3GP / 3G2. Typically recording at QVGA / VGA resolution, the 3GP format allowed compact filesizes necessary for storing video files in a time when phones didn’t have much on-board RAM (to process a video while recording / playback) or storage space (which was often not extensible on such early smartphones). As you can see in this example Tom and Jerry video, at the amount of compression used on such phones, video quality was poor and often had block distortion artefacts. Furthermore, for storage saving reasons the paired audio format used with the 3GP container was AMR or low-bitrate AAC that results in distorted audio; typically recorded mono channel or doubled-stereo from mono recording. (Never search “3GP” on YouTube. I did, to find an example, and I got pages after pages of softcore porn. Brrrr.) Another drawback of early smartphones was that they could not record at 24 fps and above. Phones as recent as my erstwhile 2009-era Nokia 5630 XpressMusic could only record at 15 fps (as demonstrated by this sample video). Such low frame rates added to the jerky, low-quality effect of mobile videos making them unusable for anything beyond sharing and viewing on other mobile devices. Things started to change around the launch of iPhone 3GS, which came with the capability to record video at 30 fps. Android and Symbian handsets launched around the same time could record at similar frame rates, with some able to do it at 720p and others at 480p. The seed for my desire to film a project on a phone was sowed by this attempt by Nokia in 2010, who commissioned a short film shot entirely on an N8 (starring Dev Patel of Slumdog Millionaire fame as well as Baywatch‘s Pamela Anderson). Fast forward to 2013, when most smartphones can record 720p video at 30 fps, typically in MP4 format. Significant advances in sensor technology, video processing software, even optical image stabilisation in higher-end phones means that the resulting video shot on phones these days is of much more decent quality. Certainly workable for amateur video projects. I expect, given the rate of progress in the field these days, that such advancements will continue to trickle down to cheaper devices. Another important aspect of any video production happens to be sound. It may not be the first thing on your mind, but good audio quality is crucial in video recording. Smartphones are getting better at this too, with multiple microphones, noise cancellation, and high-quality AAC audio recording. You can audibly hear the difference that high quality, distortion-free stereo recording can make to a video in this comparison. The Gear Now that I’ve covered my reasons for why I think present-generation video recording on smartphones is usable-enough to be used for video projects, I’ll move to my own experience. My video was shot using my Nokia Lumia 620, so your mileage may vary according to what phone...

Music discovery is broken, and Spotify Radio is part of the problem

By on Jun 24, 2013 in Tech Takes | 1 comment

I love discovering new music to listen to, but what I don’t like is reading reviews on music blogs or magazines to find them. Reading through reviews rather than listening to discover new stuff feels so 20th century to me. Music is emotional for me – as it is with many people – and I’d rather get a feel for something new myself to find whether I like it, rather than just reading words on a page. (After I find an artist I like, though, I do read up about their history and creative process on music blogs. That’s what they are good for: no matter how obscure an artist, you’re likely to find someone who has interviewed them.) My go-to source for music recommendations used to be Last.fm. I still scrobble to my profile – partly for keeping track of my music tastes over time, mostly in the vain hope that some day I will be able to unlock this data in a usable form. Last.fm remains a one-of-a-kind service for archiving music tastes; there simply hasn’t been any replacement over the years for a service like this. Ever since they stopped on-demand playback of tracks though (due to licensing issues; it was simply financially unviable for them to offer it) the platform has been less than worthless for new music discovery. Using Last.fm radio – while good at making automated recommendations based on listening history – is quite cumbersome to use as a standalone application. As it stands now, Last.fm’s discovery mechanism is primarily focussed around surfacing recommendations based on similar tracks / albums…which makes the process of going through all of them a chore. *** I have been a Spotify customer for four years now (first, Spotify Unlimited, and now Spotify Premium). It’s every music fanatic’s wet dream – an almost-limitless library of songs, available on desktop, web, mobile, offline that syncs everywhere, and for the artists holding out from adding their works to their library, it’s easy to add local MP3s to your collection. For over a year now, I was using the We Are Hunted Spotify app to source my music recommendations from. The neat idea behind We Are Hunted was that it scoured the web for signals of what music people were listening to – rather than relying merely on charts such as Billboard – and automatically created digests of what the hottest tracks in each genre were. I loved it, because for people like me who are into indie / alternative emerging music, it surfaced finds that may not even yet be climbing on any official charts yet…but thanks to Spotify’s vast tie-ups with independent labels could still be found in its library. There were times when I discovered music so underground that they weren’t available anywhere other than Spotify, or perhaps, Soundcloud (not even YouTube!). Many of my “What I Have Been Listening To” recommendations came about this way. Every month, We Are Hunted would publish an automatically-generated playlist of the hottest music and make it available through Spotify. I looked forward to this day every month with the same kind of excitement one would for a magazine issue. Then, it all ended when We Are Hunted was acquired by Twitter. Functionally, it does the same thing that it used to – in a far more cumbersome interface. Now, I need to login to Twitter Music using my Twitter and Spotify accounts, and play the tracks through that interface. There’s no easy way to export the list as a playlist any more, as the obvious intention here is to lock people in to playing music through Twitter’s own interface and / or music apps. Herein lies the problem with every other music discovery service I have ever seen: they expect the user to play part of the curation process either by selecting ‘channels’ or ‘users’ to follow (such as Hype Machine, 8tracks, and others), or by basing recommendations on individual albums / tracks / artists / playlists (such as Last.fm and countless Spotify apps). The downside, as I see it, is that either way it requires effort on part of the user to constantly prune lists of whom to follow, or in the second scenario to refresh the recommendations manually. This is a cumbersome process! We Are Hunted was doing something extremely unique with the service they were offering. It’s a service useful enough for me that I’d pay for something like it on top of my Spotify subscription fees. (The closest replacement that I have (recently) stumbled upon is Tunigo. It’s still not a perfect replacement though for We Are Hunted as its top-of-genre lists count absolutes rather than ‘rising’ tracks like WAH used to.) *** The need for such a service or an app would be obviated if Spotify itself offered an equivalent. It has moved somewhat in this direction with the launch of its new Discover feature. Apps such as We Are Hunted and Tunigo are good, but what I really want is a leanback, interaction-free music recommendation engine – much like Last.fm’s radio feature – that simply melts into the background and throws up one new track after another. Spotify does something similar with its Radio feature, on the surface, but in my experience it isn’t truly a radio service. Instead of having a native recommendation engine, Spotify integrates with The Echo Nest’s music discovery APIs to...

Netflix vs Lovefilm Instant: my impressions on streaming services in the UK

By on Nov 28, 2012 in Reviews, Tech Takes | 0 comments

I am a cord cutter. I live in a house of other cord cutters. By that, I mean that we don’t own a television in our house, instead opting to watch all our video content on our computers or portable devices. The obvious advantage of this is that none of us need to pay for a TV license. While I haven’t been able to find any market research to back this up, anecdotal evidence that I know suggests to me that a sizeable number of students are like me in that they consume a majority of their video content streaming rather than on television. The UK is different from the US television market in that a lot of the original TV content produced here is made available online quickly. There’s BBC iPlayer, Channel 4’s 4oD, and ITV Player – from the top three British broadcasters – that make their content available for free-of-cost (without a subscription or a TV license). The exception is Sky, which uses exclusivity of its content as a unique selling point for its own services. What remains then, for the viewing needs of cord cutters in the UK is a) streaming movies b) on-demand playback of older content no longer active on the broadcasters’ own services. This is the gap that online streaming services such as Netflix or Lovefilm Instant fill in. Now, I could easily pirate the content but I these days I try to do the right thing. I buy all my books from the Kindle store rather than pirating PDFs. I subscribe to Spotify‘s premium service. I genuinely believe that content creators deserve to be compensated for their work. (Reading Paul Carr and Monday Note has convinced me that digital content needs to be paid for to create sustainable businesses which will continue to amuse and entertain us.) So even with films and television, I want to get my content from legal sources rather than pirating them. There simply isn’t any excuse when the price for such services is so affordable: £5-6 a month is something that can comfortably fit into any student’s budget. Although I’ve been a subscriber to Spotify’s premium service for years, it’s only recently that streaming video services have launched in the UK have become mature enough to use. I only make an exception when there’s no legal avenue at all to obtain something I’d happily pay for; that’s when I pirate. I totally understand why they don’t make it available because at the moment, they don’t want to cannibalize their existing business. Even making it available on a paid basis gives an incentive for people to cancel their satellite / cable subscription, and that revenue it far to worthy for them right now to risk it to online services which is a more price sensitive market and won’t accept higher prices. (Read this Fast Company piece on the struggle Hulu is facing in the US along these lines.) What’s good for the customer is that most streaming service companies offer month-long or longer free trials that give you a fair amount of time to test how good their service is. This is exactly what I did. Here are my thoughts on the ones I tried out. Lovefilm Instant Lovefilm Instant (£4.99 / month) is Amazon’s play in streaming services in the UK. (In the US, this is branded as Amazon Instant Video.) I tried out a 45-day free trial of Lovefilm from a voucher I got along with an Amazon purchase. My first impression of the service was that it’s a confusing mess. DVD / Blu-Ray titles are mixed in with streaming titles. The ‘Instant’ bit essentially lives on a section of the main Lovefilm site. Discovery is primarily done through ‘lists’ created within Lovefilm according to genre and lists created by users such as ‘Best of Lovefilm’ or ‘Staff picks’. This feels odd. I remember a time, many months ago, when Lovefilm also used to make films available for a payment and some included within the subscription package, so the whole ‘With Package’ section these days – when it no longer offers films on payment – feels like they tried to stuff the current titles into the old interface. The search function does not have autocomplete. You’re flying blind as to what’s available and what’s not – or if you don’t know how to correctly spell a film title or actor / director name. The hangover of the legacy business of renting DVDs becomes quite apparent whenever you search for a title: results thrown up show a mix of DVDs as well as streaming titles. If I’m a streaming-only customer, why make things more complicated by showing me results that I cannot possibly access on my subscription plan? Perhaps this is a ploy at upselling you to their costlier plans, but for someone like me who doesn’t even have a DVD drive, this is completely pointless. While it’s possible to filter the search results according to ‘lists’ again to show only Lovefilm Instant titles, my point is a user shouldn’t have to do this extra step themselves. Okay, so let’s say you don’t have a particular film in mind and just want to browse titles they have according to genre. So I clicked on a list, and to bring some sanity into sifting through the results, choose the option to order results according to ‘Member rating’....