Cocoa Touch: Working With Image Data

Images in Cocoa Touch, represented by the UIImage class, are a very important subject. Apple’s iOS platform prides itself on visual appeal, with Retina Displays, custom UI in many top apps, and a focus on photos with apps like Instagram. To that end, it behooves you as an iOS programmer to know a bit about working with images. This post won’t discuss everything you need to know about using the UIImage class, as that’s more appropriate for a book than a blog post—though maybe a series of blog posts would do—but instead will focus on one advanced topic: working with pixel data. You can find the basic stuff in the UIImage documentation, anyway.

Turning an Image Into Data

One of the first things you might want to do with a UIImage object is to save it to disk. To do that, you’ll need to save it to an image file. There are built-in functions to get properly-formatted data from an image, in both PNG and JPEG functions:

  • UIImagePNGRepresentation(), which returns an NSData object formatted as a PNG image, taking a pointer to a UIImage object as its sole parameter.
  • UIImageJPEGRepresentation(), which returns an NSData object formatted as a JPEG image. Like the previous function, its first parameter is a pointer to a UIImage object, but it has a second argument: a CGFloat value representing the compression quality to use, with 0.0 representing the lowest-quality, highest-compression JPEG image possible, and 1.0 representing the highest-quality, lowest-compression image possible.

Once you have the image data represented by an NSData object, you can then save it to disk with various NSData methods, such as -writeToFile:atomically:.

Getting Raw Pixel Data

While the above functions are great for saving images, they aren’t so great for image analysis. Sometimes you need to analyze the pixel data for a given pixel, down to the values for the red, green, blue, and alpha components. To get that kind of granularity in an image, we’ll be using a lot of CoreGraphics functions. If you haven’t used CoreGraphics before, know before going in that it’s a C-based API à la CoreFoundation, so you won’t be using the Objective-C objects you know and are used to. Instead, there are opaque types (represented by CFTypeRef, which is analogous to Objective-C’s id) representing objects grafted onto C, complete with manual memory management—no ARC for you. That’s neither here nor their, however; let’s talk about pixel data.

Color Space

The color space of an image defines what the color components of each pixel are. Represented by the CGColorSpace type, you’ll typically use either an RGB color space or a Gray color space, which have red, green, and blue components or a white component, respectively. For this example, we’ll be using the RGB color space. We can create an instance of it with the CGColorSpaceCreateDeviceRGB() function, which returns a CGColorSpaceRef type—think of it as a pointer to a CGColorSpace object.

What does using this color space get us? We now know that the pixels of our image will have three color components, and in what order. This will come in handy later on when we need to query the data.

Graphics Contexts

A graphics context, represented by the CGContext type, is analogous to a painter’s canvas—it’s what you draw into. For the purposes of drawing an image, you’ll create a CGBitmapContext, the ideal type of context for this data. You create a context with the CGBitmapContextCreate() function, which return a CGContextRef type. Let’s look at the declaration of that function (from CGBitmapContext.h):

CGContextRef CGBitmapContextCreate (
    void *data,
    size_t width,
    size_t height,
    size_t bitsPerComponent,
    size_t bytesPerRow,
    CGColorSpaceRef colorspace,
    CGBitmapInfo bitmapInfo
);

So, that’s pretty simple, right? It’s actually fairly straightforward, despite its appearance. Let’s break it down into more easily-digestible components. It’ll make more sense if we don’t go top-to-bottom, so we’ll go in the order I think makes the most sense.

First is the bitmapInfo parameter. The CGBitmapInfo type is a bitmask that represents two options: the alpha component, which contains transparency information, and the byte order of the data. We’ll talk about the alpha component here; byte order is another topic altogether. On iOS, only some pixel formats are supported. Looking at this chart in the documentation, we can see that, for all supported pixel formats on iOS in the RGB color space, these are the CGBitmapInfo constants we can use:

  • kCGImageAlphaNoneSkipFirst
  • kCGImageAlphaNoneSkipLast
  • kCGImageAlphaPremultipliedFirst
  • kCGImageAlphaPremultipliedLast

We can do two things with the alpha component: skip it, or use it in a premultiplied format. The premultiplied flag tells the system to multiply the individual red, green, and blue components by the alpha value when storing it. So, instead of RGBA values of 1, 1, 1, and 0.5, it’s stored as 0.5, 0.5, 0.5, and 0.5. This is a performance-saving measure on iOS devices, and is done automatically to all of your PNG images by Xcode when you build for a device.

So, for the bitmapInfo parameter, I generally pass kCGImageAlphaPremultipliedLast.

The penultimate parameter, colorspace, is a CGColorSpaceRef pointing to a color space you’ve created. This informs the context about the number of color components. Keep in mind that there’s one extra component for the alpha information if you’re not skipping it, so an RGB color space uses 4 components including alpha.

The width and height parameters are pretty simple: the number of pixels wide and high to make the context. Keep in mind that for Retina displays, you may need to double the values. You can use the scale property of the main UIScreen object as a quick “am I on a Retina device?” check.

Next, let’s talk about the first parameter: data Here you have two options: to pass in a pointer to a region of memory you’ve allocated for the image data, or to pass NULL and have the graphics subsystem create it for you. If you’re trying to access pixel data, however, it’ll help to have a pointer to the data, so here you’d pass in memory you’ve allocated. How do you know how much is enough? Let’s look at the bitsPerComponent parameter. I usually use 8-bit components—again, see the chart linked above for valid options—so I would pass 8 for bitsPerComponent. Once you know that, you can determine bytesPerRow easily:

size_t bytesPerRow = (bitsPerComponent * width) / 8;

And then, finally, we can determine how much data to use. I use the uint8_t data type to represent this, as it’s an unsigned 8-bit integer, perfect for our needs.

uint8_t data = calloc((width * height) * numberOfComponents, sizeof(uint8_t));

The entire stack might look like this:

The only thing in this code that we haven’t gone over so far is the call to CGContextDrawImage, which (surprisingly) draws the image. It takes three parameters: the context to draw into, a CGRect defining where to draw, and a CGImageRef for the image. You can obtain a CGImageRef from a UIImage using its -CGImage method.

Now that the image is drawn in our context, the rawData array will be filled with real, live image data! You can access it like so (modify the values of x and y as suits your needs):

int x = 0;
int y = 0;

int byteIndex = (bytesPerRow * y) + (x * bytesPerPixel);

uint8_ t red   = rawData[byteIndex];
uint8_ t green = rawData[byteIndex + 1];
uint8_ t blue  = rawData[byteIndex + 2];
uint8_ t alpha = rawData[byteIndex + 3];

And there you have it! Now that you’ve gotten the data out of your image, do whatever you want with it. Just remember the blog authors you read along the way when Facebook buys you for a billion dollars.

Note: The venerable Mike Ash published a similar article while this one was half-done in my drafts folder. I thought about scrapping it altogether, but since mine is iOS-specific, and with some prodding from a co-worker, I decided to press on. Go read Mike’s blog, too. It’s awesome.

Cocoa Touch Tutorial: Using Grand Central Dispatch for Asynchronous Table View Cells

One of the problems that an iOS developer will often face is the performance of table view cells. Table view cells are loaded on-demand by the UITableView that they’re a part of; the system calls ‑cellForRowAtIndexPath: on the table view’s dataSource property to fetch a new cell in order to display it. Since this method is called (several times) while scrolling a table view, it needs to be very performant. You don’t have very much time to provide the system with a table view cell; take too long, and the application will appear to stutter to your users. This kills the immersion of your application and is an instant sign to users that the application is poorly-written. I guess what I’m saying is that this code needs to be fast. But what if something you need to do to display the table view cell takes a long time—say, loading an image?

In my MobiDevDay presentation a couple of weeks ago, I illustrated a solution to this problem: Grand Central Dispatch. GCD, Apple’s new multiprocessing API in Mac OS X Snow Leopard and iOS 4, is the perfect solution for this problem. Let’s take a look at how it works.

Grand Central Dispatch operates using queues. Queues are a C typedef: dispatch_queue_t. To get a new global queue, we call dispatch_get_global_queue(), which takes two arguments: a long for priority and an unsigned long for options, which is unused, so we’ll pass 0ul. Here’s how we get a high-priority queue:

dispatch_queue_t queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_HIGH, 0ul);

It’s pretty straightforward. To use this queue, we add blocks of code onto it. Typically this is done with blocks (Apple’s new code encapsulation extension to the C language), though it can be done with C functions. To submit a block onto a queue for execution, use the functions dispatch_sync and dispatch_async. They both take a queue and a block as parameters. dispatch_async returns immediately, running the block asynchronously, while dispatch_sync blocks execution until the provided block returns (though you cannot use its return value). Here’s how we schedule some code onto a queue (we’ll assume this code runs after our previous example, so queue is already defined):

dispatch_async(queue, ^{
    NSLog(@"Hello, World!");
});

It’s very easy to forget the ); at the end of that line, so be careful.

How does this apply to table view cells? Let’s take a look at a typical scenario for loading images from disk:

- (UITableViewCell *)tableView:(UITableView *)tableView
         cellForRowAtIndexPath:(NSIndexPath *)indexPath
{
    static NSString *CellIdentifier = @"ExampleCell";
    
    UITableViewCell *cell = [tableView dequeueReusableCellWithIdentifier:CellIdentifier];
    if (cell == nil) {
        cell = [[[UITableViewCell alloc] initWithStyle:UITableViewCellStyleDefault
                                       reuseIdentifier:CellIdentifier] autorelease];
    }
    
    // Get the filename to load.
    NSString *imageFilename = [imageArray objectAtIndex:[indexPath row]];
    NSString *imagePath = [imageFolder stringByAppendingPathComponent:imageFilename];
    
    [[cell textLabel] setText:imageFilename];
    UIImage *image = [UIImage imageWithContentsOfFile:imagePath];
    [[cell imageView] setImage:image];

    return cell;
}

The problem with that code is that creating image blocks until ‑imageWithContentsOfFile: returns. If the images are especially large, this is catastrophic. Modifying this code to use Grand Central Dispatch is simple:

- (UITableViewCell *)tableView:(UITableView *)tableView
         cellForRowAtIndexPath:(NSIndexPath *)indexPath
{
    static NSString *CellIdentifier = @"Cell";
    
    UITableViewCell *cell = [tableView dequeueReusableCellWithIdentifier:CellIdentifier];
    if (cell == nil) {
        cell = [[[UITableViewCell alloc] initWithStyle:UITableViewCellStyleDefault
                                       reuseIdentifier:CellIdentifier] autorelease];
    }
    
    // Get the filename to load.
    NSString *imageFilename = [imageArray objectAtIndex:[indexPath row]];
    NSString *imagePath = [imageFolder stringByAppendingPathComponent:imageFilename];
    
    [[cell textLabel] setText:imageFilename];
    
    dispatch_queue_t queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_HIGH, 0ul);

    dispatch_async(queue, ^{
        UIImage *image = [UIImage imageWithContentsOfFile:imagePath];
    
        dispatch_sync(dispatch_get_main_queue(), ^{
            [[cell imageView] setImage:image];
            [cell setNeedsLayout];
        });
    });
    
    return cell;
}

First, we create our image asynchronously by using dispatch_async(). Once we have it, however, we have to come back to the main thread in order to update our table view cell’s UI (all UI updates should be on the main thread, unless you like reading crash reports). GCD has a function to get the main queue—analogous to the main thread—called dispatch_get_main_queue(). We can dispatch a block to that thread to update the UI.

By making this simple modification, we can very easily improve the performance of our table view. There are a few steps remaining, however, and this method has one serious shortcoming: if the cell is re-used by the time the image loads, it can load the wrong image into the cell. To get around this, it would be better to cache the images in an array or a dictionary (just be sure to release it in your view controller’s ‑didReceiveMemoryWarning: method). That said, this is an example of something you can do quite easily to improve the performance of your application. The better it performs, the more your users will like it, and that’s the ultimate goal.

The code used in this post is available as a GitHub repository.

Cocoa Touch Tutorial: Stripping Non-Alphanumeric Characters on Entry in a UITextField

In a previous post, I showed you how to trim non-alphanumeric characters from a string. Here I’ll go more in-depth and show a method that I wrote to restrict text entry in a UITextField to alphanumeric characters. Since I also wanted the characters to be uppercase, I’ll also ensure that only uppercase characters are allowed.

This should all happen in the - (BOOL)textField:(UITextField *)textField shouldChangeCharactersInRange:(NSRange)range replacementString:(NSString *)string method of your UITextField’s delegate (which, of course, must implement the UITextFieldDelegate protocol). I’ve implemented it as follows:

- ( BOOL )textField:( UITextField * )textField
shouldChangeCharactersInRange:( NSRange )range
  replacementString:( NSString * )string
{
    /*
     * We only want uppercase letters and numbers in this text field, so if
     * this method is adding something else, we don't want it. But we also
     * want to support copy-and-paste, so it's not always going to be one
     * character added.
     */
    BOOL shouldAllowChange = YES;

The shouldAllowChange variable is set to YES initially because we want to allow this change when possible. The method will test the string to see if it meets criteria for rejection as we move forward.

    NSMutableString *newReplacement =
    [[ NSMutableString alloc ] initWithString:[ string uppercaseString ]];
    
    if ( ! [ string isEqualToString:newReplacement ]) {
        shouldAllowChange = NO;
    }

First, we define newReplacement. It’s an NSMutableString so that if we discover non-alphanumeric characters in it, we can remove them on-the-fly. It also serves as a convenient string against which we can test to see if string is already uppercase.

    NSCharacterSet *desiredCharacters =
    [ NSCharacterSet alphanumericCharacterSet ];
    
    for ( NSUInteger i = 0; i < [ newReplacement length ]; i++ ) {
        unichar currentCharacter = [ newReplacement characterAtIndex:i ];
        
        if ( ! [ desiredCharacters characterIsMember:currentCharacter ]) {
            shouldAllowChange = NO;
            [ newReplacement deleteCharactersInRange:NSMakeRange( i, 1 )];
            i--;
        }
    }

In this section, we define the NSCharacterSet that we want to work with - in this case, the alphanumeric character set. We go through one character by a time and if the current character isn’t alphanumeric, we remove it from the NSMutableString (decrementing i so that we don’t inadvertently skip a character) and set our shouldAllowChange flag accordingly.

    if ( shouldAllowChange ) {
        [ newReplacement release ];
        return YES;
    } else {
        [ textField setText:[[ textField text ]
                             stringByReplacingCharactersInRange:range
                             withString:newReplacement ]];
        [ newReplacement release ];
        return NO;
    }
}

To finish, if shouldAllowChange is still true, we return YES and allow the replacement characters to be added. Otherwise, we return NO, but not before using our replacement replacement string (say that ten times fast) to manually edit the text field’s text. The end result is a text field that will consist only of uppercase letters and numbers.

Cocoa Tutorial: Strip Non-Alphanumeric Characters from an NSString

Let’s say you have an NSString that contains both alphanumeric and non-alphanumeric characters and you want to strip the non-alphanumeric characters out of it. The hard way is to manually go through, character-by-character, and put the character in a new string if it matches certain criteria. But why do it the hard way?

Apple provides a class that we can use for this to great effect: NSCharacterSet. We want alphanumeric characters, so we can create a character set of the characters we want using this method:

NSCharacterSet *alphanumericSet = [ NSCharacterSet alphanumericCharacterSet ];

Now we have a character set like we want. We just need a way to turn our string into a string that contains only those characters. Unfortunately, the closest thing in NSString’s implementation is the -stringByTrimmingCharactersInSet: method. But that seems to do the opposite of what we want. Fortunately NSCharacterSet has our back here. We can use the -invertedSet method. So here is our final code:

NSString *beginningString = @"Some string with non-alphanumeric characters. !@#$%^&*()";
NSCharacterSet *nonalphanumericSet = [[ NSCharacterSet alphanumericCharacterSet ] invertedSet ];
NSString *endingString = [ beginningString stringByTrimmingCharactersInSet:nonalphanumericSet ];

In this example, endingString will be equal to “Somestringwithnonalphanumericcharacters”.

UPDATE: As it turns out, this only works if the non-alphanumeric characters are at the beginning or end of the NSString. Whoops.

Cocoa Touch Tutorial: Extract Address Book Address Values on iPhone OS

This is the first of what I hope to be several Cocoa Touch tutorials on this site.  I was doing some furious Googling last night trying to find out how to get a contact’s street address from the Address Book for an upcoming update to Take Me Home, and I realized that it’s complicated and there aren’t any good tutorials online.  So, after I figured it out, I commented it up so that hopefully, if you’re reading this, you’ll save some time that I didn’t.

Before you read this tutorial, you should go through Apple’s excellent Address Book Programming Guide for iPhone OS.  This tutorial will rely on the QuickStart application you write in the guide, so do that first.
The first thing we need to do is add an address field to the QuickStart application.  Use Interface Builder to add a new UILabel underneath the two you already have.  You may want to stretch it to fill the entire width of the screen, like so:

Add a new UILabel underneath the exisiting two.
Add a new UILabel underneath the exisiting two.

Now, add the information about this label to QuickStartViewController.h:

//
//  QuickStartViewController.h
//  QuickStart
//

#import <UIKit/UIKit.h>
#import <AddressBook/AddressBook.h>
#import <AddressBookUI/AddressBookUI.h>

@interface QuickStartViewController : UIViewController <ABPeoplePickerNavigationControllerDelegate> {
    IBOutlet UILabel *firstName;
    IBOutlet UILabel *lastName;
    IBOutlet UILabel *addressLabel;
}

@property (nonatomic, retain) UILabel *firstName;
@property (nonatomic, retain) UILabel *lastName;
@property (nonatomic, retain) UILabel *addressLabel;

- (IBAction)showPicker:(id)sender;

@end

Be sure to go back into Interface Builder and connect File’s Owner in QuickStartViewController.xib to addressLabel.

Now, we have to change the method that gets called when you click on a person in the ABPeoplePicker.  As it is at the end of the QuickStart tutorial, once you select a person the picker is dismissed.  So, we do the following in QuickStartViewController.m:

- (BOOL)peoplePickerNavigationController:(ABPeoplePickerNavigationController *)peoplePicker
      shouldContinueAfterSelectingPerson:(ABRecordRef)person {
    NSString *name = (NSString *)ABRecordCopyValue(person, kABPersonFirstNameProperty);
    self.firstName.text = name;
    [name release];

    name = (NSString *)ABRecordCopyValue(person, kABPersonLastNameProperty);
    self.lastName.text = name;
    [name release];

    [self dissmissModalViewControllerAnimated:YES];

    return YES;
}

Note that you have to delete the line that dismisses the modal view controller; if you don’t, the people picker is dismissed before you have a chance to get the address. When you delete it, the people picker will continue when you select a person. Next up, we have to write the method for when someone selects an address on the next screen. Here’s the method:

- (BOOL)peoplePickerNavigationController:(ABPeoplePickerNavigationController *)peoplePicker
      shouldContinueAfterSelectingPerson:(ABRecordRef)person
                                property:(ABPropertyID)property
                              identifier:(ABMultiValueIdentifier)identifier {
    // Only inspect the value if it's an address.
    if (property == kABPersonAddressProperty) {
        /*
         * Set up an ABMultiValue to hold the address values; copy from address
         * book record.
         */
        ABMultiValueRef multi = ABRecordCopyValue(person, property);

        // Set up an NSArray and copy the values in.
        NSArray *theArray = [(id)ABMultiValueCopyArrayOfAllValues(multi) autorelease];

        // Figure out which values we want and store the index.
        const NSUInteger theIndex = ABMultiValueGetIndexForIdentifier(multi, identifier);

        // Set up an NSDictionary to hold the contents of the array.
        NSDictionary *theDict = [theArray objectAtIndex:theIndex];

        // Set up NSStrings to hold keys and values.  First, how many are there?
        const NSUInteger theCount = [theDict count];
        NSString *keys[theCount];
        NSString *values[theCount];

        // Get the keys and values from the CFDictionary.  Note that because
        // we're using the "GetKeysAndValues" function, you don't need to
        // release keys or values.  It's the "Get Rule" and only applies to
        // CoreFoundation objects.
        [theDict getObjects:values andKeys:keys];

        // Set the address label's text.
        NSString *address;
        address = [NSString stringWithFormat:@"%@, %@, %@, %@ %@",
                   [theDict objectForKey:(NSString *)kABPersonAddressStreetKey],
                   [theDict objectForKey:(NSString *)kABPersonAddressCityKey],
                   [theDict objectForKey:(NSString *)kABPersonAddressStateKey],
                   [theDict objectForKey:(NSString *)kABPersonAddressZIPKey],
                   [theDict objectForKey:(NSString *)kABPersonAddressCountryKey]];

        self.addressLabel.text = address;

        // Return to the main view controller.
        [ self dismissModalViewControllerAnimated:YES ];
        return NO;
    }

    // If they didn't pick an address, return YES here to keep going.
    return YES;
}

Let’s go through that in more detail.  The method gives us the following information: an ABRecordRef of the person we’ve selected, an ABPropertyID of the property slected (in this case, we ensure that it’s the address) and an ABMultiValueIdentifier of which address we’ve selected.  It is important to note that the ABPropertyID is equal to kABPersonAddressProperty when you select any address; that is, there is only one address property. This one address property holds the values in an ABMultiValue, each at a specific index.  Here are the steps we take in the code:

  1. The first thing we do is define our ABMultiValue, multi, and copy the contents of the selected value into it.
  2. Then we define an NSArray, theArray, into which to copy the multiple values.  But which one do we want?
  3. Each address has an identifier, which the method gives to us as identifier, but we reference them by index when getting them out of the array.  So, we need to create an index (which we’ll store as an unsigned integer), theIndex, and set it to the return value of the ABMultiValueGetIndexForIdentifier function.  Now that we have the index, we know which value of the array to store .  They’re stored as type CFDictionary, which have key-value pairs for us to use, so we define an NSDictionary, theDict to put them into.
  4. First, we need to know how many key-value pairs there are, so we use the count method and store the return value in an unsigned integer, theCount.  Be sure that this variable doesn&rquo;t change—you don’t want to assume that there are more members in the array than there actually are, as that can lead to nasty memory problems. For that reason I’ve defined it as a constant.
  5. Now, we define two NSString arrays, keys[theCount] and values[theCount], and then we’re ready for action.
  6. Next we use the NSDictionary getObjects: andKeys: function to copy the keys and values. The function copies the data, and we can construct our street address.  For the purpose of this example, I’m going to make the address a single line, but you do with it what you want.
  7. Finally, we create a final NSString to put the formatted address into, pull the values out of the dictionary into the appropriate place, and we’re all done!

Update 2010-07-27: Removed [ theDict release ];, some bad memory management.