Saturday, 28 March 2020

QGIS on Ubuntu 18.04

QGIS didn't work for me out of the box. Here's notes on what I had to do to get it going. Brief summary, out of the box the Open Street Map URL is missing. See 'Breakthrough' below for how to fix this. However, also, the version of qgis available from the Ubuntu package repository is at present quite out of date; alternative repositories are available but as yet I haven't tried them.

Firstly, if it isn't working and you've been tinkering to try to get it working, start by purging it completely:

sudo dpkg --purge qgis qgis-provider-grass qgis-plugin-grass
rm -rf ~/.qgis2/

Then reinstall it with all the extra bits that it actually needs - when I originally installed it, it did not automatically install saga, and I'm not sure whether it automatically installed the grass plugin.

sudo apt install python-qgis qgis-plugin-grass saga

This at the time of writing installs QGIS version 2.18.17.

Then start QGIS


QGIS, immediately on starting.
At this point the application window renders. I get one python warning in the 'Log Messages Panel':
2020-03-28T08:08:03 1 warning:/usr/lib/python2.7/dist-packages/qgis/ DeprecationWarning: This method will be removed in future versions. Use 'parser.read_file()' instead.

Because it's only a deprecation warning, I'm inclined to think this is not a significant issue.

I also get the following warnings in the console. Once again, they're only warnings:

Warning: loading of qgis translation failed [/usr/share/qgis/i18n//qgis_en_GB]
Warning: loading of qt translation failed [/usr/share/qt4/translations/qt_en_GB]
Warning: Object::connect: No such signal QgsMergedBookmarksTableModel::&QgsMergedBookmarksTableModel::selectItem( const QModelIndex &index )
Warning: Object::connect: (receiver name: 'QgsBookmarksBase')
Warning: QCss::Parser - Failed to load file "/style.qss"
QInotifyFileSystemWatcherEngine::addPaths: inotify_add_watch failed: No such file or directory
Warning: QFileSystemWatcher: failed to add paths: /home/simon/.qgis2//project_templates
Warning: QLayout: Attempting to add QLayout "" to QgsPanelWidgetStack "mWidgetStack", which already has a layout

I select, from the menus, Project -> New; the 'Recent Projects' heading disappears from the main pane, and it appears to be a map pane - but there's no map.

I type 'world' into the 'Coordinate' input and hit return, as directed by this tutorial, and still, no map. Comparing my screen to the screen shown in the tutorial, they are identical except that 
  1. No map is displayed, and
  2. Some icons in the second toolbar from the top which are coloured in the tutorial are greyed on my screen.
I take this to mean that not only can I not see the map, QGIS doesn't think there is a map from which I can make selections.

If I try to drag the 'osmraster' item from the 'Browser Panel' into the map area, I get a series of errors in the 'Log Messages Panel' of the general form:
2020-03-28T08:37:53 1 Tile request max retry error. Failed 3 requests for tile 11 of tileRequest 2 (url: https:/2/2/2.png)

Clearly, the URL is malformed. Clearly, there is something wrong with the template string from which the URL is being formed.

At this point I tried using the 'Getting Started' instructions from the manual. Once I had downloaded the sample data, I was able, following instructions, to get raster data to show on the map pane. Vector data, however, for example from the 'lakes.gml' file in the sample data, still did not render at all.

This made me suspicious of the error seen earlier in the console,

Warning: QCss::Parser - Failed to load file "/style.qss"

If the default style is white on white, then perhaps QGIS has been rendering a map, but rendering it invisibly. I find that there are indeed two files with this name in my file system:

simon@mason:~/qgis$ locate style.qss
/home/simon/.qgis2/themes/Night Mapping/style.qss
/home/simon/simon/.qgis2/themes/Night Mapping/style.qss
/usr/share/qgis/resources/themes/Night Mapping/style.qss

However, I can't find any way of selecting a stylesheet in any of the menus (there's a 'Style Manager' dialogue, available from the 'Project Properties' dialogue and from the 'Settings' menu, but it does not mention stylesheets and does not appear to allow you to set one. There's no reference to 'style.qss' in any file in my .qgis2 directory; nor is there any reference to it, or to a stylesheet at all, in a saved project file.

I experimentally copied one of these style.qss files into the root directory of my filesystem, and restarted QGIS. This time, the file was found, but its only effect was to render many dialogues unreadable; it did not change the blankness of the default map panel. Further browsing showed that a QSS stylesheet is a stylesheet used by the Qt tookit (which QGIS is built on), not used by QGIS itself, and, as the default styles render QGIS perfectly acceptably, this file is not needed and is not the problem.

At this point I noticed what looked like specks of dirt on the screen. I zoomed in and found a group of vector shapes which could well be lakes - but they didn't AT ALL align with the sample raster data. I checked the coordinate systems used and found that they were different. I changed the coordinate system for the lakes data to that used for the raster data, and they overlaid precisely. Progress!

The breakthrough

So I went back to trying to find why the Open Street Map data was not loading. I right clicked on 'Tile Server (XYZ)' in the 'Browser Panel' and got a dialogue asking for a URL. I entered '{z}/{x}/{y}.png', chosen from an existing leaflet project, and was shown another dialogue asking for a name for the layer. I entered 'TestOSM'.

A 'TestOSM' entry appeared in the 'Browser Panel', and when I dragged this to the map pane, suddenly I had a map.


My takeaway from this catalogue of problems is that the QGIS package offered by Ubuntu is 
  1. Misconfigured, and
  2. Obsolete
I did get it working, and I've documented what I needed to do above; but it's clearly some way behind the current 'stable release', which is 3.10.4.

So I've spent a whole morning trying to install a more up-to-date version of QGIS. There are many repositories for more recent versions of QGIS out there on the Web, and many 'howto' articles explaining how to set them up and use them. I've tried many this morning, and my conclusion is that they're all, without exception, broken in one way or another. Some people do seem to have succeeded in getting a working install, but even their notes don't work. 
Furthermore, after doing that, it took an awful lot of work to sort out the mess and get back to a working 2.18.7. In summary, I believe it can be done, but it's not for the faint hearted.

Thursday, 27 February 2020

Putting data on the map

A couple of weeks ago, someone came to me with a problem. She had data in a spreadsheet. She wanted to display it as a map, on a website. And she wanted to be able to do that dynamically - that is to say, she wanted the map on the website to update as the spreadsheet changed.

So there were clear routes to several of the parts of this problem:

1. If you need maps on a web page, you need Leaflet - it's a simply wonderful library;
2. If you want a spreadsheet that is live on the Internet, Google Sheets are a very good way to go;
3. Google Sheets allows export as Comma-Separated Values, and CSV is a very easy format to parse.

I needed to do this quickly, so I started in my comfort zone, using Dmitri Sotnikov's Luminus framework and mixing in the Day8 re-frame framework, in order to have a lot of built-in functionality client side. The joy of the Clojure development tools is that getting a working system is as easy as typing

    lein new luminus geocsv +re-frame

And from that point on you can do your development with your system live, and can see the effects of every change you make immediately.

So within a couple of days I had a polished wee system which looked good and pulled live data from a Google Sheets spreadsheet to populate a map, with different graphical pins for her different classes of records. It was highly reusable, since you can specify the document id of any public Google sheet in the query part of the URL, and that sheet will be rendered. The map automatically pans and zooms to focus on the data provided. Was I pleased? Well, sort of.

Why was I not pleased? The deliverable component is an executable jar file, and it's 58 megabytes; in the browser, client-side, a page consumes 10 megabytes of memory. That seems huge for such a simple piece of functionality. I'd satisfied the requirement, but I hadn't satisfied myself. Also, of course, it needed you to be able to run a component on your web server, and many organisations with simple web hosting can't do that.

Doing it again, better

So I started again. This time I did it entirely client-side, just ClojureScript, with no heavyweight libraries. Having solved the problem once, it was pretty easy to do it again, and within a day I had a system working. Furthermore, I added flexibility: you can supply a URL, as before; or as the text of the document element that the map overlays; or passed as a string to the function. You can see it here.

Was I pleased?


Three things about this solution don't satisfy me.

Firstly, the deliverable (jar archive, for direct comparison with the original) is still 2.9 megabytes. That's only 5% of the size of the original, and includes the whole of Leaflet, the whole of [Papaparse]( - a client side CSV parser, and 398 different map pin images, but it still seems big. Page load - with the three maps on the demo page - costs around 5 megabytes in the browser.

Secondly, because it's client-side only, it cannot know what pin images are available on the server; so if there is a category value in the data for which no pin image is available on the server, you will get a 'broken image' appearing on the map, which is ugly.

Thirdly and most significantly, again because it's client side only, modern cross-site scripting protections mean that it cannot pull data from a spreadsheet hosted on another site - so it doesn't strictly meet the original requirement.

Iterating again

I decided to look at whether I could make it smaller by abandoning the comfort of the Clojure environment and writing pure JavaScript. This led to a third iteration, geocsv-js, which you can see in action here. The amount of 'my code' in this version is far lighter - the output of the ClojureScript compiler for geocsv-lite comes to 6.6 megabytes uncompressed, whereas the pure JavaScript version is one file of just 8.8 kilobytes in 296 source lines of code (for comparison, the ClojureScript solution, geocsv-lite, comes to 372 source lines of code). One reason for the slightly larger size of the ClojureScript solution is that it has a better algorithm for panning and zooming the map to display the data actually entered, which I didn't port to JavaScript.

But what is bizarre and I do not yet understand is that the deliverable is not smaller but bigger than the geocsv-lite deliverable, by almost twice - 4.7 megabytes - despite using identically the same libraries. Still more surprising, the memory consumption in the browser is also higher, at around 6 megabytes.

What are the lessons learned? Well, the overhead of using ClojureScript is not nearly as much as I thought it was. There is something clearly wrong about the discrepancy between the size of the packaged deliverables - the pure JavaScript variant must be including third party data which the ClojureScript variant doesn't, although I haven't yet tracked down what this is - but the in-browser memory footprint is actually smaller, and the page load time is, as best as I can measure, identical at about five seconds.

Was I satisfied now? Well, sort of.

Merits and demerits

All three variants have merits. The first pulls the data directly from Google Sheets, which was the original requirement, and served a default pin image for any record category for which it didn't have a matching image. That made it a lot slicker to use, and more forgiving of user error. The two client-side-only variants cannot do those things, for reasons which I have not found ways to solve. But they don't need any server-side functionality beyond the dumb serving of files, so they're much easier to deploy; they are also less greedy of client side resources.

One more time with feeling

One of the reasons why I kept on hammering at this problem was that I felt it would make a really useful extension for my wiki engine, Smeagol. I've integrated the JavaScript version, geocsv-js, and in doing that I've solved a number of problems. Firstly, the look and feel and content of Smeagol pages is flexible and easily configurable by the user, so it doesn't take a geek to set up a page with the content you want and a map of the data you want to show. Secondly, the Smeagol engine, because it sits server side, can pull data from remote sites, and because it doesn't interpret that data, there's no significant risk in doing that. Thirdly, again because it sits server side, it can deal with the issue of unknown images - and, because it's a wiki engine targeting less technical users, I've deliberately made it very graceful about how it does this.

So now, instead of just a map on a web page, you get a whole, richly editable website, with existing extensions to integrate sophisticated data visualisation and photograph galleries as well as maps. And the cost of this? Surprisingly little more. A Smeagol page with one map uses exactly the same memory on the client as a geocsv-js page with one map, because Smeagol now only loads JavaScript for extensions actually being used in the page, and almost all of its own functionality runs server side. But even server side, the cost is not very much greater than for the full fat geocsv implementation - the deliverable jar file, which offers far more functionality than geocsv, is only 88 megabytes. Considering how much more usability and flexibility this offers, this is the version of geocsv I'd now offer, if someone came to me with the same problem.

Creative Commons Licence
The fool on the hill by Simon Brooke is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License