Now that we've had a look at common issues with IIIF servers and simple ways to detect most of them, let's have a look at the many ways we worked around those issues within our product, Arkindex. We will go quite in-depth, in the hopes that this blog article may help our users and instance administrators better understand what happens under the hood.

While client projects and organizations can document the previously-discussed issues, try to raise awareness about them, or work with server administrators to resolve them, some clients will still rely on images provided by non-compliant servers. In addition, we can never know in advance when an IIIF server will introduces new issues or become mission-critical. Therefore, IIIF usage in Arkindex has been centered around a few goals:

  • Avoid using less commonly used IIIF features such as rotation, mirroring, or non-default image qualities, which are rarely tested and often have more issues;
  • Give explicit error messages to the user when an issue is detected with an IIIF server or when Arkindex does not support an IIIF feature;
  • Automatically correct issues client-side whenever possible to ignore a server's mishaps;
  • Make it easy for Arkindex instance administrators or developers to perform any necessary manual actions to work around server issues.

Image management

Elements in Arkindex are not directly tied to IIIF image URLs. Instead, we have two separate concepts: images and image servers. These concepts enable us to handle most of the issues related to each image's info.json, as well as missing or moved images.

To be clear, image servers in this context are not actual servers: they are rows in a database table. An image server in Arkindex holds some attributes to link it to an actual IIIF server, and all of the image management we do here is managing the links between Arkindex and external IIIF servers. Even an IIIF server that is maintained purely for Arkindex itself will be treated like any external IIIF server.

Image servers

Each image is linked to an image server. Images hold the identifier part of the IIIF image URL, and image servers have the hostname and prefix. When we need to build an image URL, we concatenate the server's URL with the image identifier, then add the crop, resize, rotation and quality parameters. This separation is visible within the Arkindex API:

>>> cli.request('RetrieveElement', id='591478c0-fce7-43c7-b8a0-ce21bb39fff7')
{
"id": "591478c0-fce7-43c7-b8a0-ce21bb39fff7",
"type": "page",
"name": "Demo element",
"zone": {
"id": "591478c0-fce7-43c7-b8a0-ce21bb39fff7",
"url": "https://server.example.com/iiif/my-image/0,0,2000,1000/full/0/default.jpg",
"image": {
"id": "23eb3e0b-2033-4c4f-ad52-eae49c49486f",
"path": "my-image",
"width": 2000,
"height": 1000,
"status": "checked",
"url": "https://server.example.com/iiif/my-image",
"server": {
"display_name": "Demo Server",
"url": "https://server.example.com/iiif",
"max_width": None,
"max_height": None
}
},
"polygon": [
[0, 0],
[0, 1000],
[2000, 1000],
[2000, 0],
[0, 0]
],
},
# ...
}

The zone attribute on elements is an historical artifact, as we used to have a concept of Zones between Elements and Images; in reality, both the Image UUID and polygon are directly stored on an Element. Both the zone URL and the image URL are dynamically generated during the API request: the image URL is a concatenation of the server's URL and the image's path, while the zone URL is a full IIIF Image API URL to retrieve the portion of the image that contains the element's polygon.

These various ways of using the image's URL mean that an API consumer can choose how much they want to care about IIIF. They can:

  • Trust the Arkindex backend to give them a full IIIF URL in zone.url;
  • Add their own Image API parameters after zone.image.url;
  • Build their own URLs from scratch using zone.image.server.url and zone.image.path.

This makes it easier to use IIIF for simple cases while still enabling more complex customizations. The width and height properties also mean that it is not necessary to retrieve the info.json file for each image before using it: the Arkindex backend will retrieve the info.json file once and cache the most important properties. This reduces the amount of HTTP requests made to an IIIF server, and helps prevent accidental "denial of service" attacks since many servers can quickly crumble after numerous image requests.

Moving images around

One of the issues we mentioned in the first post was that images could sometimes be moved to other URLs without notification. Sometimes, the change can be as simple as switching the server from HTTP to HTTPS, or as complex as switching to another domain name. When such a change occurs, the split between images and image servers in the database enables Arkindex instance administrators to make a single update: changing the server's URL from http://... to https://... will immediately affect all of the server's images that are known to Arkindex, allowing it to keep up with the changes on the IIIF server.

For more complex cases, such as two different servers being merged, a single server split into multiple, or all images in a folder being moved, an administrator can use the server splitting and merging features to update the stored image URLs in bulk and match the new URLs. We extended the Django administration interface to add two split and merge buttons on each server:

  • Splitting a server requires typing in a folder name. A new server is then created, which holds all the images in this folder. For example, splitting https://server.example.com/iiif on a folder named a would result in a new server at https://server.example.com/iiif/a, and all the images whose paths began with a/ will now be in this new server without the a/ prefix.
  • Merging two servers requires both servers to have a common URL. For example, merging https://server.example.com/iiif/a and https://server.example.com/iiif/b will result in one server at https://server.example.com/iiif, and the image paths will be updated to begin with a/ or b/ respectively.

These two features alone allow Arkindex instance administrators to handle most of the bulk image move operations. For example, if some images were moved from /old to /new on the same server, it would be possible to apply this move in three operations from the web interface:

  1. Split the server on /old.
  2. Change the new server's URL so that it points to https://server.example.com/iiif/new.
  3. Merge the original server with the new one.

While this can sound like a roundabout way to handle moving images between folders, this system proved to be more flexible than only providing a "move" button to move images between folders. Additionally, while a move like this would be doable without too much hassle directly using SQL queries on the Arkindex database, merging or splitting servers can be much more complex and bug-prone. Having just a few buttons to click saves time and effort.

Image status

While having the ability to update URLs on thousands of images quickly is helpful, automatically detecting when images have gone missing is important too. To do so, and to detect numerous other potential issues on each image, we introduced the concept of image status. An image begins as unchecked, and can then either be in a checked or an error state.

Arkindex provides two main ways to request a check on an image:

  • The arkindex check_images command for system administrators, which can check every known image or only a small sample;
  • Updating the image's status to checked using the PartialUpdateImage endpoint. This will first run the check before applying the update: if the check fails, the status will change to error instead.

When an image's status is set to error, our web interface will display a warning:

The API reports issues with this image from the IIIF server at https://server.example.com/iiif. Navigation, annotation and processes may have unexpected behavior.

Explicitly displaying in our web interface that the issues stem from the IIIF server, instead of showing nothing or showing a less readable error message generated by the web browser, can help make our users more autonomous. They might know themselves how to reach the administrator of that particular IIIF server and ask them for help, or will at least be able to report to their instance administrator or to Teklia that something is wrong.

The image check process tries to perform an Image Information Request for the image, which means it will retrieve the image's info.json file. If retrieving this file fails, we know something is wrong with the image.

But the image check does not stop there and tries to parse the file. For a check to be fully successful, the following conditions must be met:

  • The info.json file must be accessible and readable as JSON;
  • The file should include a profile property with an IIIF profile URI;
  • The width and height properties must be valid positive integers;
  • The image's identifier (@id) must begin with the server's URL.

The image check process is able to detect when an image's path has been updated, as long as the @id begins with the server's URL: it will automatically update the image's path as necessary. This will however fail if another image with this path already exists.

Similarly, the check can also update an image's width and height to reflect any changes in size. All element polygons on this image will be resized to fit these new dimensions. This makes it possible to handle cases where an IIIF server administrator upgrades the images to higher resolutions, for example.

Maximum width and height

The last part of the image checking process includes managing the concepts of maximum width and height. In the IIIF Image API, it is possible for a server to specify the maximum size of any image that it can return: this enables server administrators to limit bandwidth and memory usage for each request. This however means that the typical /full/full/0/default.jpg URL may no longer return the image in its entirety but may instead result in a resized image.

Arkindex currently only supports the maxWidth and maxHeight properties. While maxArea is commonly found on many IIIF servers, this setting is usually defined at a value much higher than the combined maximum width and height, which has made it unnecessary for our use cases.

The image check process will try to look for the maximum width and height in two different places, to support some particular cases of IIIF non-compliance:

{
"@context": "http://iiif.io/api/image/2/context.json",
"@id": "https://server.example.com/iiif/my-image",
"width": 2000,
"height": 1000,
"profile": [
"http://iiif.io/api/image/2/level0.json",
{
# Compliant definition of maximum width and height
"maxWidth": 4000,
"maxHeight": 4000
}
],
# Non-compliant definition found in some servers that were critical to our projects
"maxWidth": 4000,
"maxHeight": 4000
}

When the correct, standard definition is not found, Arkindex will fall back to looking for the attributes on the image object itself instead of the IIIF profile.

If either the maximum width or height are found, Arkindex will compare those values to the max_width and max_height attributes defined on the image server. If those values are different, then the check will fail, notifying an instance administrator. The administrator can then compare with other images and update the image server's settings to match.

IIIF defines no way of accessing a server's global settings, and nothing in the Image API specification states that maximum widths or heights are global to a server and not specific for each image, but in our experience, all servers either have the same maximum size settings for all of their images, or no defined maximum size at all. Either way, if multiple sizes are found on the same server, then splitting the server can be the solution to support different settings.

These maximum width and height attributes are not used by the Arkindex backend at all. They are only managed in the image check process to be made available in the API, letting API consumers use other features such as tiling to request images should they need images larger than those size settings.

IIIF imports

Outside of the API's own image management features, various other features of Arkindex use the API to either declare IIIF images or use them. The main way to import images from an external IIIF server is through the IIIF manifest import.

This import task supports both collections and manifests from the IIIF Presentation API 2.1. As such, its main concern will be over the issues of the Presentation API and not the Image API.

Fetching remote data for JSON-LD

We mentioned that JSON-LD's complexity can lead to numerous issues, and its Linked Data aspect has security and reliability concerns. For those reasons, our IIIF import task does not support fetching remote data anywhere in an IIIF manifest or collection, with a single exception: IIIF collections can list URLs pointing to manifests, and those will be downloaded. Our experience has been that most data providers will not try to use linked data anywhere other than for linking to manifests inside of collections, since this tends to be confusing and does not fit the typical examples shown in the Presentation API specification.

Multi-valued properties

Many properties of an IIIF manifest, such as a page name, a manifest description, or metadata values can use JSON-LD value objects, which can be strings, objects or arrays. Arkindex does not support multi-valued strings like these, so the import task uses this algorithm to find out which value to use:

  • If the value is a primitive (a string, an integer, etc.), use it directly.
  • If the value is an object, use its @value property. If there is no @value, cause an error.
  • If the value is an array that does not contain only JSON-LD value objects, cause an error.
  • If the value is an array of JSON-LD value objects:
  • Use the @value property of the first object that has an @language property set to English.
  • If no object is found that way, just pick the first object in the array.
  • If there is no @value in the selected object, cause an error.

For metadata values, since Arkindex does support creating multiple metadata with the same name, a different process is followed:

  • If the value is a primitive (a string, an integer, etc.), use it directly.
  • If the value is an object, return its @value property. If there is no @value, cause an error.
  • If the value is an array:
  • If all the array items are JSON-LD value objects and have an @language property, follow the same process as above for language detection.
  • Otherwise, treat each item as a separate value and run each through this algorithm.

The whole IIIF import task has gone through multiple iterations and this algorithm has proven to have the highest success rate on the data that we acquired.

Detecting image servers

The concept of an image server means that to create a new image, Arkindex first has to know on which server this image should be. When importing external IIIF manifests, it is common to get new unknown servers. For this reason, Arkindex implements automatic server detection in its API.

Since the IIIF Image API does not define the concept of an image server at all or how a server should behave at its root URL, it is not possible to know with absolute certainty where the division between the server part and the image identifier part is in an image URL. Making API requests on every subfolder in the URL until we found one that appeared to be successful does not work: https://server.example.com/iiif can return anything, from a 404 error to a very large HTML response that we do not want to deal with in the backend.

Our automatic detection therefore works by using common IIIF server prefixes: for example, every Cantaloupe server will provide IIIF 2.x support at /iiif/2/, and most servers will use /iiif/ since this is the prefix used in most examples of the API specification. When we spot an unknown server, if it has one of the common prefixes, we will immediately make the split and create a server automatically. The instance administrator may later have to use the image server splitting and merging features to reorganize as necessary.

If none of the well-known common prefixes are found, then the image creation fails, and an error message is shown in the logs instead to invite the user to request that an image server is created for them.

Displaying images

The Arkindex frontend is where most users will interact, perhaps unknowingly, with IIIF servers directly, and where most of the issues will be found and reported. Therefore, displaying IIIF images in this frontend requires particular care to show explicit errors, perform checks that we might only be able to do after requesting real images, etc.

Cropping and resizing

The frontend will use the width of the browser window to determine at which size it will request images: this means that browsers on smaller screens will need less bandwidth to load images, and hopefully that the memory usage on the IIIF server is reduced.

When we know that the maximum size we will accept is greater than the image's own size, instead of applying an unnecessary resize parameter, we will just use full: some unoptimized servers cannot deduce that a resize is unnecessary, or may make the image larger than its original version. The maximum size on a server is largely ignored, since we are already ready to handle images that are smaller than expected due to our own resizes.

Similarly, while we usually crop requested images to fit the element's polygon, if it appears that the polygon matches the entire image, we will not crop at all.

Checking dimensions

When opening the details on an element with an image, the image will be the most important part of the user interface. This is the most important place in which to tell the user that the image has an issue of any kind, as annotating on an incorrect image could lead to a lot of wasted work. Therefore, ominous red error messages may appear above the image to strongly warn the user if they are about to step on a landmine.

In addition to the message shown on an error status as mentioned above, the frontend also checks that any image that it requested has the expected size. This does take into account that the server may have a maximum size. If the image is of a different size, then it is very likely that either the server does not support the few IIIF features that Arkindex requires, or is not compliant with the specification and returning invalid images. We therefore show a warning:

The size of the provided image is 500×250, but 2000×1000 was expected. There may be an issue with the IIIF server at https://server.example.com/iiif.

Rotation and mirroring

Release 1.1.0 of Arkindex added two new attributes on elements to specify their rotation angle and a horizontal mirroring. These attributes map the rotation parameter of the Image API 2.x specification, as it includes both clockwise rotation and horizontal mirroring. However, many servers do not tolerate rotation and mirroring well: there can be inconsistencies between implementations or between each image, HTTP errors, or just no rotation at all if the server does not support it and does not want to cause an error.

Therefore, we have not used IIIF to provide rotation and mirroring. The element annotation screen instead uses CSS transforms to have the browser rotate and mirror the image and re-maps polygon coordinates by itself. This was definitely non-trivial and there still are bugs, but this re-implementation of an IIIF feature works more reliably than using the IIIF feature itself.

IIIF rotation and mirroring has however been used directly for element thumbnail URLs, displayed in element lists. Those thumbnails are less important than the real image, and the cost of implementing the same rotation and mirroring all over again is far greater than the impact of some servers potentially not rotating properly.

Tiling

As of writing, we do not use the IIIF tiling features, even though they usually work better than other features due to their use in most IIIF viewers such as Mirador. This might however appear in a future version of Arkindex, to help handle very large images on which zooming in to view precise details is necessary and for which an image server is unable to provide everything in one large image.

Using images in Machine Learning workflows

The final important use of IIIF in Arkindex is in Workers. In all workers that are based on our base-worker template, a helper method is available to open an element's image directly without concerning oneself about the details of IIIF: Element.open_image(). This method does the smartest IIIF image URL handling in Arkindex, as it can use tiling whenever necessary.

When the server has no maximum size, open_image will do a regular cropped request, similarly to the frontend's own behavior. But when the server has a maximum size, then the info.json file is requested. It is assumed that the file's tiles property is required in this case to provide the recommended tile sizes. The full image is then downloaded by requesting the largest available tiles, using an algorithm similar to the one offered as an appendix to the Image API specification.

This way of loading the image is important for Machine Learning workflows as using the highest possible resolution means that workers can make full use of the image in any way that they want. Most Machine Learning models will require the Python code to first resize the image to a rather low resolution, and sometimes turn it into grayscale, but using the full resolution at the start means that no additional artifacts will be added from multiple resize operations, which can have an impact on the error rate of a whole workflow.

Conclusion

This post has shown you most of our uses for IIIF and how we handle the many specification compliance or complexity issues of the IIIF server landscape. We hope this enables our advanced users and administrators to better understand the how and the why of IIIF handling in Arkindex, and maybe prompts more in-depth discussions with IIIF servers administrators that would like to be fully supported with Arkindex.

While we hope that we would be able, in the long term, to remove some of these workarounds and make better use of all the features IIIF can offer, it is likely that we will have to keep them to stay compatible with the few servers that would still not change. We can at least hope that those workarounds will become less and less necessary over time, reducing the amount of errors that users have to cope with when using external servers with Arkindex.

In the last installment of this series of posts on IIIF troubles, we will look at how IIIF 3.0 takes into account some of the issues we've encountered and the pros and cons, from our point of view, of how they've been mitigated.