Exasperation

Right, having started the New Year on the high of a long-awaited publication, it's time to get back to reality.

Which is, and I can't stress this enough, dismal.

I've done goodness-knows how many posts about politics... oh wait, I made a tag for that, so let's count 'em. That's 47 here, which tend to be quite long and (dare I say it) detailed, and a whopping 1,026 over on Decoherency. Which tend to be shorter, more current, and a lot more ranty.

Some topics deserve a fuller treatment though. And the Current State Of The World Right Now is just such a thing. You can't call yourself a blogger and not cover the return of the Orange One. You just can't.

They take away your blogging privileges if you do. Although maybe it'll be the opposite. At the rate social media is going, perhaps soon you'll have to write panegyrics for flatulent orange blubberbags instead.

Anyway, as I was saying, I've written a lot about politics, because it is after all a thoroughly interesting subject. After politics, they say, everything is ashes (perhaps that might turn out to be more literally true than the usual interpretation). But more importantly, I've tried, God knows I've tried, to understand why people believe apparently incomprehensible things and support people who say things which are dumber than a bag of particularly dumb rocks, and I realise that... I've failed.

So to warn you, this is not a post in which I'm going to attempt much in the way of explanation. Oh, there'll probably be a few passing comments here and there, because sometimes I just can't help myself : after all, we have to try. But for the most part this is going to be no more than an emotional outburst, a paean to exasperation, a eulogy for common sense and decency*. It will also use a lot of intemperate language. Readers of a nervous disposition, pregnant ladies, and those with a heart condition should look away now. Probably.

* Not quite entirely though, because anyone who tells you that everything is hopeless and we're just innately a shitty species is a cynical loon who shouldn't ever be taken seriously.

America, Where Art Thou ?

Let me start with the US's apparent determination to shoot itself in the foot and a good many other places besides, what with the US being the isotropic home of gun-toting crazies. To be honest I got lucky. When the election result came through I was away on conferences and there just wasn't time to worry about it. Yet the sheer bizarreness of it all still hasn't sunk in. Kamala Harris had such a blisteringly good start to the campaign, surging in the polls and slipping only slightly later on, that I never really worried much about the outcome. It seemed almost a foregone conclusion.

For the opening credits of Have I Got News For You, they used to have a cartoon of Obama effortlessly getting the basketball in the net. Later they changed it so he missed.

Which of course it wasn't. The Atlantic has it that Democratic losses weren't truly catastrophic for the party; perhaps there was more an element of not voting at all rather than actively shifting right. This seems very much to have been the strategy of election manipulators in the past, though I've yet to see a detailed statistical analysis of these results. 

What gives me a lot of pause for thought on that possibility is the multitude of reports of Trump supporters being incredibly unenthused : falling asleep in or walking out of his rallies en masse (though I don't know of anyone who quite managed to do both and sleepwalked out), polls showing remarkably little energy or support for the campaign. And conversely, Harris seemed to have consistently massive rallies; the "you want the other smaller rally down the street" moment was genuine gold, as was the success of calling the Repubnicans "weird". That painted a very rosy picture of apathy among the MAGA crowd and a determined enthusiasm among the liberals to get out and vote*.

* Note that I'm not saying that the strategy of firing up one's own base by denigrating the opponents is the right approach. I'm just commenting here on the apparent differences between the two sides, with one appearing markedly more successful than the other.

I mean they hardly look wild with excitement. Couple that with many pictures of stadiums half-empty, plus multiple polls showing that Trump supporters were doing little more than going through the motions, and the general trend seemed clear.

For whatever reason, it didn't turn out like that. I just have two comments on that point. First, I heard from someone (but haven't been able to find the story online) that one pollster won millions of dollars by betting on Trump because he asked people not who they'd vote for, but who their friends would vote for. Apparently that gave a much more decisive result. People might well be too embarrassed to say they themselves would vote for the Oompa Loompa in-chief (psychologically, confessing even in an anonymous poll may still be embarrassing) but less so about admitting what their friends would do. And we know that if you do have many friends who share a belief, you're very likely too as well. So that one works both ways, being both an accurate reflection of what people believed but also because of who was, by their own admission, persuading them.

Secondly, there was one clip – I think it was on John Oliver – contrasting two particular campaign events. The Harris one on this occasion was in a church, very solemn, dignified, serious, and... boring. The Trump event was unhinged, with a crazed Elon Musk jumping like he's on some implausibly potent drugs and Hulk Hogan turning up for who the fuck knows why, probably there were cheerleaders shooting bald eagles with fireworks (or the other way around*) and all the other crazy American stuff as well. I dunno. Unhinged to be sure, but you couldn't possibly call it boring. And it stuck in the back of my mind ever since that the Trump event was, if I may abuse the stereotypes, a lot more American-feeling than the Harris one.

* Or the other other way around. Fireworks shooting bald eagles with cheerleaders.

That may have course been an outlier though. Most of Harris' events looked a lot more lively. But it did worry me, a bit.

Perhaps I was caught in a filter bubble ? It's very possible. Maybe there were more enthusiastic Trump supporters elsewhere, maybe Harris alienated more people than it appeared. Maybe there were selection effects : you don't need much enthusiasm to vote, and of course by definition the existing fanbase are bound to turn up in legions at the rallies. So Harris' huge crowds may not have reflected the wider voting community. Maybe. We similar effects all the time; after all, nobody takes opinion polls on specific issues very seriously because they're generally so unrepresentative. 

Regardless, what I think everyone has utterly failed to get a handle on is the most basic issue of all. We can do detailed statistical breakdowns to the nth degree, but this doesn't answer the fundamental problem : why in the hell anyone listening to Trump for more than about 30 seconds doesn't feel an uncontrollable urge to a) punch him very hard indeed in the face and/or b) run screaming down the street, their brain oozing out of their ears in an effort to escape the unbearable nonsense it has to put up with.

That is... I genuinely think that we have some pretty good insights into how persuasion works in groups. That obscenely long linked post is not, mercifully, a complete waste of time. But while we also know a good deal about how persuasion works among individuals, I feel like we're still missing a huge piece of the puzzle. We don't understand the fire-is-hot problem, how people can experience the same evidence and come to strikingly different conclusions that one would naively think equivalent to claiming that fire is cold instead of hot. I'll quote myself :

I don't know how to persuade anyone that fire is hot if they won't believe it when their own hair is actually on fire. Those kind of errors are supposed to be dealt with by the Darwin Awards, not persuasion.

As I've said, I've tried. I keep records of how I've changed my mind. I've tried to self-assess how I make my own political decisions. I've looked at why bullshit is appealing. I've read books about persuasion and rhetoric. I've read numerous threads on social media attempting to explain why liberalism isn't appealing to people and none of it is much help. 

I don't say there's no value in any of this, but... when I hear someone saying, "I'm going to be a dictator" and a host of other statements to the same effect, and worse besides, I can't not care about it*. I can't somehow assume this was said purely for dramatic effect or some such nonsense. I immediately assume that the kind of person who would say such things, for any reason whatever, is in no way fit to be near any political office of any kind. Either they're truthful, in which case they want to assume power over me and mine which I don't want them to have, or they're lying and are therefore not to be trusted.

* Along with a legion of other insane remarks, not least on health and foreign policy. America First ? You deplorable selfish bastards. You don't think it matters to say things like Russia should be given a free hand in Europe, eh ? Apparently not, with one BBC article quoting Trump supporters who said it was all about the price of eggs. The selfish fucking cunts.

Trump supporters appear not to care about this. That Trump is demonstrably a liar doesn't bother them one bit. Exposing his lies, important though it is, has absolutely no effect on them at all because that simply doesn't interest them. And I find that absolutely baffling, in the extreme. That's the root of the problem, the bit I think nobody has addressed : how in the holy flying hells are there so many people who think like this ? How can they vote for someone when they know they can't trust him to do what he promises – if, that is, they can even string a coherent message out of his verbal diarrhea at all ?

A small cult of loonies is to be expected. A few nutcases here and there ? Sure, people are people, weirdos gonna weirdo. Whole societies in the past were incomparably worse than voting for degenerate racists. But a huge chunk of a modern industrialised nation ? And again, I'm not talking about the kind of reasoning that involves a robust statistical analysis or deep critical thinking skills. I'm talking about problems almost literally at the level of realising that fire is hot (I'm not using hyperbole here), that if you elect someone who promises to all but eat your babies, then this – FFS – isn't a good idea. 

What leads me to absolute and profound exasperation is that this is the kind of thing that even with such fecal disseminations as Fox "News"* and a plethora of social media outlets, you absolute bunch of brain-dead fucktards can't see the most buggeringly obvious thing... even when it's hurtling towards you from behind in a truly worrying fashion with a pronounced bulge and a menacing grin. It doesn't matter what your surroundings are like, you should be able to reason for and by yourself. You should be able to realise that Man Who Want Eat Babies Bad, Woman Who Don't Want Eat Babies Better.

* Though I think the "Fox" should also be in quote marks, because it doesn't relate to foxes either.

There's probably an element of this at work but even this isn't enough for a full explanation.

And no, people of Quora, I don't buy your explanations that Me No Like Liberals Who Want Take Toys Away. Liberals are not, in the main, constantly telling you that everything you've ever loved is evil, and it takes no effort at all to understand this. Now to be fair, there are some pretty stupid memes around and plenty of activists who do rather insist on this sort of highly toxic, puritanical, holier-than-thou behaviour... but you can just ignore 'em. It's really easy ! A bunch of morons on social media saying mean things is not an excuse for voting for a frickin' would-be dictator, you absolute bunch of effluent, incontinent, impotent man-babies.

Ahem. I did warn you this was going to be a rant.

Anyway, I might be prepared to excuse some of this as due to a particularly perverse version of American exceptionalism, a strain (or stain) perhaps of their peculiar culture : an excessive veneration for material wealth, an obsession with influencers, alpha-bro culture and leadership. But I can't do that, and to see why, I turn back across the pond to my native soil.

Britain, What Are You Even Talking About ?

I watched the UK's general election with triumphant glee. And you know what's changed my mind about the result since then ? Nothing, you feckless bunch of twats. I still believe in the government every bit as much as I did before the election. The Labour government isn't perfect, because "perfect government" is an unrealisable oxymoron, and to their enormous credit they tempered expectations brilliantly on that score (if anything too much so). But it is, nonetheless, doing a bloody good job, though you wouldn't think so from either the media or the polls.

Don't worry, this section of the post will be considerably shorter than the first. The same major themes apply as above, and I already did a short write-up of the main issues here. In brief, the Labour government has done a pretty good job of sorting out the mess the country's in, considering that they've got 14 years of damage to undo and have only been in power for a few months so far. Scandals ? Negligible. The worst we've had is that they claimed some freebies and declared them properly. Now, listen up and listen good :

That's

Not 

A 

Scandal !

I don't care if it's not a good look given the economy, it's not a scandal. It's unrealistic in the extreme to expect people not to take material advantage when they're allowed to, and if you think you wouldn't do so yourself, you're likely deluded. The real scandal is the endemic Tory corruption that's being ignored because of this absolutely pathetic and pointless non-story. 

Incompetence ? Also negligible. Sorry, but means-testing pensioners for money most of them don't need so that it can be given to other, genuinely needy people, that's a good thing. So is making farmers with millions of pounds in assets pay inheritance tax on them, especially when you consider the many exemptions and that they can simply gift it to their descendants ahead of time. If these really are small family farms being affected, then as Private Eye put it, the tax is "essentially voluntary". And we've seen huge progress on reinvesting in the economy, an end to strikes, a commitment to green energy... look, this government is clearly, unarguably, going in the right direction. Of course it isn't bloody there yet you twerps, they're not wizards ! Things take time to have an effect ! Why is this hard to understand ?!??

Say what you will, I feel the AI got the mood of what I was after here pretty well.

And yet what we've seen from the media is nothing but relentless, grinding negativity. Because the PM had a free suit he's clearly in the pocket of Big Fashion. Because he used the apartment of a Labour Peer – an actual Labour party member for God's sake – he clearly somehow pocketed tends of thousands from this even though no money changed hands because why the fuck would it ? Because the budget didn't immediately cause skyrocketing economic growth the Chancellor is clearly a monstrously incompetent amateur and deserves to be put in the stockade or torn apart by bears or something.

Worse... who's benefitting from this perceived though absolutely phoney incompetence ? Not the Tories, thankfully, who are languishing the doldrums of having elected yet another bloody loony. No, it's the even stupider bunch of loonies who've had more practise at cultivating insanity : Reform, a.k.a. the Nigel Farage party. Party ? Drunken-unintelligible-old-white-men-slurring-words-in-a-pub, more like.

I just don't get it. You perceive dishonesty and incompetence among Labour, fine. You're an idiot, but fine. But to then say, "I know who'll really get things sorted out, it's Reform" is a truly special kind of stupid. I am aghast.

Part of the exasperation is not knowing what to take seriously any more. Ironically, part of this is due to Rishi Sunak's best speech : that of his departure. This was humble, contrite and respectful well beyond what was needed. It made me feel that the guy probably wasn't that bad, he's just fallen in with a bad crowd. The problem is that all this is so at odds with the rhetoric he used in Parliament that it's impossible to know who's the real Rishi (if there even is one). Is it Parliamentary performer Rishi or is that naught but a façade ?

When statements can be given which are so contradictory, making sense of anything becomes excruciatingly difficult bordering on impossible. The same goes for stated policies, especially across the pond. If the so-called leader can't actually formulate a coherent sentence, how can you ever know if he's doing what he said he would ? Yes, there's an element of theatricality to political performances, but when rhetoric loses meaning to this degree, what's the point of listening to anything ? At the moment accusations from the right are verging on the point of calling Keir Starmer a werewolf or a banana cunningly disguised as a lawyer : it's such abject nonsense as to rot the brain of the viewer.

Where Do We Go From Here ?

Goodness only knows. For the UK the situation is nowhere near a state of despondency; exasperation is all that's warranted at the moment. Polls this early in a government don't mean much. Give the policies time to actually take effect and it may be a very different story a year or so down the line – but only if the government gets its act together on messaging. Clearly, they've been objectively bad at that. As things stand Labour have a thumping majority and aren't afraid of taking unpopular decisions. This is good, because they're going to need to force things through to effect real change.

But they can't rely on a promised upturn forever. One of the big problems with not knowing what to take seriously is just how likely a Reform breakthrough would be. For now the answer is "not very", but if there's a single lesson from this, it's that we have to take the batshit stuff seriously and can't rely on people's common sense. They don't have any.

Which is hugely depressing. That people were willing to keep voting for the Tories and their absolute drivel, their corrupt incoherence and rampant corruption, their hollowing out of the state for their own aggrandisement, for 14 years, but then decide almost instantly that because Labour haven't fixed everything right away and aren't literally saints that they must somehow be just as bad.... well, sod the whole dang bleedin' lot of you. You're not making any sense. It makes me want to physically scream.

Some hope is gleaned from the in-fighting currently afflicting the far right. This is true in America as well, though there things are undeniably in a worse state since the crazies have (for now) won. Success is not final, failure is not fatal, and all that. I take some comfort from Trumps' utter untrustworthiness, who is at least not a raving ideologue like some of his so-called cabinet. Slim pickings indeed.

I'm not sure I can venture much of an explanation for any of this. Best guess : it's a combination of insular communities with self-reinforcing beliefs and a media who are singularly bad at remaining objective. Criticising everyone without giving praise where praise is due just makes everything seem awful the whole time. This is not really how a proper critique is supposed to be done : it's like the news is on permanent "debunking" mode rather than "verification". Couple this with rampant sensationalism and disaster ensues. What's the point of even trying to do the right thing if the whole press are just going to rip you to shreds over the most minor details regardless ? This environment is a hotbed of stupidity. The result, especially in America, is a cult on a grand scale.

There's a nice quote in Amazon's underrated sci-fi drama The Rig :

The truth is not enough ! Not anymore. It hasn't been for a long time. Behind me... behind everything... there is a vast engine of power, dedicated to making sure that the future you want does not happen. That we never learn, never change, never stop. Something this big, even with proof, people have to choose to believe it. The people we're fighting don't give up. 

In the show it's a conspiracy. In reality it's nothing of the sort, it's right out in the open. That's the tragedy of the whole thing, that we can see what's going on, that we have the means to tell everyone... and it doesn't work.

I don't know what to do about this. I know lots of people think they do, but I can't get over the fire-is-hot problem. People wanting to watch the world burn is one thing, but when they themselves feel the lick of the flames and do nothing... the old meme is more pertinent than I thought. If the artist would like to suggest a solution, I'm all ears.

You Can All Get FRELLED

New Year, New Software

How to start the blogging for the New Year ? Some topics are obligatory, like the current, shockingly abysmal state of global politics. Don't worry, I'll get to that. But I prefer to start on a more optimistic note, and with commendable timing, I had a paper accepted on New Year's Eve*. So let's do that instead.

* Presumably due to a drunken intern at an office party accidentally pressing the "accept all" button.

Presenting the FITS Real-time Explorer of Low Latency in Every Dimension. From the Farscape pseudo-swearword meaning a sort of combination of "hell" and "fucked". Hence the title of this post is extremely insulting.

The physics of galaxies is a jolly interesting thing, but you can't very well study what makes galaxies tick without having a sample of galaxies to start with. Finding the gassy buggers can be a right pain in the backside, so my major side-project is developing better ways to detect them. Sometimes that means writing fully automatic code that turns data directly into catalogues, but I prefer the old-fashioned approach of actually looking at the data we spend some much time and effort collecting. 

My main visualisation effort has been a code called FRELLED, which long-term readers will know is something I've been working on for a very long time. Basically it turns everyone's favourite 3D art software, Blender, into something astronomer-friendly.

... or at least, that was the idea. In practise... eeeehhhh, not so much. When it worked, it did work. You could load a 3D data set and fly around it and catalogue sources and everything ! It was great. I couldn't understand why it hadn't gone viral and made me millions of pounds from a grateful astronomical community. 

Two reasons. The obvious one is that astronomers prefer to spend their millions on telescopes and suchlike, rather than giving random donations to obscure Welshmen who give their codes away for free. The second, more interesting reason is that when FRELLED didn't work it, err, didn't work. It was pretty versatile in some ways but hopelessly limited in others. The learning curve was better than learning Blender from scratch but still pretty steep, it was full of bugs, and calling it astronomer-friendly would be overly-generous. It was astronomer non-hostile at best. 

It's very easy to provoke astronomers, mind you. Especially with clouds.

Lockdown finally provided me with the incentive to undertake the formidable task of rewriting FRELLED's ~11,000+ lines of Python code to bring it into the modern world. And not just because it needed an overhaul, which it did. But also because it had finally become simply unsustainable. With appalling timing, the earlier version had been written for Blender 2.49, which became obsolete almost  immediately as the very next version of Blender totally rewrote all of Blender's Python syntax*. This was hugely irritating : just as I'd got the hang of Python in Blender, they went and rewrote the whole bloody thing. I had the choice of either sticking with the functioning code I had or immediately starting from scratch.

* Python users – this was not like the upgrade from Python 2 to Python 3, which was relatively minor in comparison. No, this was a complete overhaul, so that virtually every internal Blender Python command was replaced with something totally different.

Well ! Naturally I chose the former, since I do have to do actual science as well at some point. I don't have the luxury of spending 100% of my time developing software – I need to actually use it. So FRELLED persisted in an abandoned version of Blender for several years, frequently tinkered with, never fully renovated.

But then lockdown happened, and recoding FRELLED was a much-needed project that could easily be done from home. So off I went and started coding. It helped that I'd already familiarised myself with more recent Blender versions for other projects.

This took quite a while.

Then, it took a while longer.

Things got derailed at various points because of the need for other projects.

But eventually, with the code doubling in size to 20,000+ lines and a bunch of new features, it was done ! Hurrah ! Let joy be unconfined, let there be dancing in the streets.

And so we all got wasted and did tonnes of cocaine or something. Although to give an idea of how long these things take, the paper alone took six months from submission to acceptance. That's after spending a full six months just figuring out the right journal to submit to.

Or at least, it was done enough. A project like this never really ends, there are always bugs to squash and features to add. But it reached a point where it did everything I wanted to do with a satisfyingly low number of bugs that one wouldn't need a flyswatter to turn it on.

I already did a somewhat in-depth look at how FRELLED works more than ten years ago (!) so I'm not going to go through it all again. Here I'll just look at what you can do with it and why you'd want to bother. If you're looking to actually use the code, the best place to get started is the home page, wherein you can find links to the embarrassingly large wiki, a complete set of video walkthroughs, and a 45-minute I-TRAIN video as well. It's got documentation in spades, and of course, you can always get in touch with me for help.

What Does It Do ?

This. It does this :

The main thing it does is look at 3D data cubes inside Blender. If you want the nitty-gritty on this see the earlier blog post, but the basics are pretty simple. Radio observations of galaxies often generate 3D data : as well as position and brightness on the sky, we also get velocity. That's not the same as distance, but using it in place of distance makes it easy to see all of our data all at once. And numerical simulations also create 3D data sets, including ones where the third axis really is distance.

The old version of FRELLED had a lot of manual coding in it because Blender 2.49 simply couldn't access a lot of more modern Python modules. That was hugely inefficient and wasteful, because Python is the equivalent of Ms Universe for astronomical codes : immensely popular. It's increasingly rare to have write your own code for basic astronomical tasks – chances are there's a Python tool for most operations. Not the high-level stuff, but for the low-level everyday processing, you're better off spending a little time searching for someone else's code than writing your own. Which makes it relatively easy to cobble all these snippets together to do the larger task you want them to do.

"Here it is, looking weird !" FRELLED is probably the short one with lots of tentacles.

What this means is that the old FRELLED had a lot of minor but annoying problems when looking at data of anything except 21cm emission. The new version, by using modules which do all the tedious coordinate calculations for me, lets it look at any wavelength with minimal fuss (at least for radio frequencies, at any rate). And cubes now load faster, sometimes very much faster indeed, and it can load data sets up to 27 times larger than the old version. Actually I've got a promising technique to increase this by maybe another order of magnitude in the works... but that's on the backburner for now.

There's more than one way to skin a cat, and there's more than one way to view a cube. Volumetric rendering, as above, is where you see the whole data set all at once. It's by far my favourite method but it's not always the best choice. So FRELLED also supports viewing cubes slice-by-slice, as it did before, but also with a couple of fun new additions. First, it can also render data slices as height maps (you might remember this from a solarigraphy conference) :

This is a nice example because it's a cube I've looked at a whole bunch of times and yet one of the major features escaped me completely. It's the galaxy M33, and in a standard map, you see the colours saturate at high values – in this case shown in red. The problem is that the dynamic range of the data is huge : the brightest features are much, much brighter than the faintest, maybe by a factor of a thousand. Choosing a colour scheme that can show features at all levels is really tough.

Same data set with the same colour scheme, but with different vertical stretching for the height maps (linear on the left, logarithmic on the right).

Height maps provide a nice way around that. As well as using the conventional colour map, each pixel is displaced by an amount according to its value. You can use different ways of mapping the data to colour and displacement (height), so in effect you're showing the data with two colour schemes at once. For example, as here, you could choose the colours to highlight the faintest features, while using the height to examine the structure in the brightest bits of the data. In this case, I'd never noticed the twin peaks in the galaxy, even though they're the dominant features !

Another major new feature is the ability to display the data as isosurfaces. These are like contours but in 3D, showing surfaces of constant data value. I have to say I originally poo-poohed these as a cheap excuse for 3D rendering : sure, it's easy to display a few thousand polygons as a surface, any fool can do that... what about rendering the hundreds of millions of voxels of the full data set, eh ? Honestly I really only put them it because I could. But I began to realise that I'd been far too harsh, because sometimes less is more. By reducing the data to this sort of very carefully, quantifiably selected display, all of the intervening noise is cut out completely. Ironically, the key to finding the most interesting features can be to remove most of the data. That goes well against my training and instincts as someone interested in the faintest objects, but it works.

And finally... virtual reality ! I love VR, I think it's one of the best technological developments of recent years. Yes, even if Mark Zuckerbot is an obsequious twat who's blatantly pandering to far-right despotism... a lot of people seem to confuse the worst sort of techbros (who are awful) with the tech they develop (which can, sometimes, be a genuinely good thing). But I digress.

Anyway, Blender 2.79 doesn't support real-time VR, so FRELLED comes with some export scripts to convert the files into a format that can run in later versions, tested up to 4.0. So you can step around your data either as isosurfaces or volumetrically, as though it were in the living room with you because why the hell not you crazy person for even asking.

Actually, I was a bit skeptical that VR would prove to have any scientific justification. I wanted this feature solely because it's cool, and if more people developed things because they were cool then the world would be a happier place*. Whether it had any real benefit was entirely besides the point. 

* Or perhaps not.

But actually, I have to say there probably is real value to this. It goes like this :

• Viewing data in 2D, slice by slice, takes a long time to build up an intuitive sense of its full 3D structure.
• Viewing the data as a 3D volume you can rotate on a 2D screen helps develop that intuitive sense much faster, because you can see all the data at once.
• Viewing the data in true stereoscopic data helps even more. Now you don't even need to rotate, making it even easier to grasp what's a coherent structure in the data and what's just a chance alignment (so-called projection effects). You instantly see the 3D structures directly.

There's even another program, iDaVIE, which is fully in VR with almost no 2D interface at all. I like it very much (and the performance is oodles and oodles better than FRELLED), though for me the analysis tasks I need aren't quite there yet. Which brings me on to the next section.

Why Should You (Not) Use It ?

You should use it because you're a cool person who loves awesome things ! As the wiki says, it's as much fun as a pile of kittens, and what right-thinking person doesn't love a pile of kittens ? Nobody, that's who.

Well... maybe I exaggerate. A bit. But it should be, I hope, reasonably user-friendly. It certainly won't yell at you or call you rude names in Welsh like the old version did. There is, unavoidably, a learning curve, but it's probably now (almost) as shallow as I can make it. 

Felix Stoehr coined this wonderful term "fastronomy", which is basically something I've mentally labelled as the Facebook Principle for many years. The referee wouldn't let me keep this analogy in the paper, but at its heart, Facebook is little more than glorified email. You could, in principle, set up a vast email list of thousands of strangers and send them pictures of your cats and/or genitalia, or rant at them about how cancer is causing vaccines or why pigeons are all mind-controlling robots. Nobody does any of that by email; lots of people do some of the above on Facebook. It's even considered normal*.

* I dunno why the referee objected. I barely even mentioned genitalia in the paper, let alone robot-pigeons.

Why is that ? Probably because Facebook et al. make otherwise awkward social interactions convenient and casual. And this is something astronomers have been traditionally very reluctant to acknowledge. As discussed at length in Malta, we often focus so much on accuracy that we forget about interface, making mistakes so easy to make that the result is inaccuracy – the very thing we were trying to avoid ! So fastronomy is something I buy into enthusiastically : we should make features as easy and accessible to use as we possibly can.

Surely this is just self-evident ? Yes and no. There is a school of thought that says some things should be challenging and hard to do. And to an extent they've got a point. Some tasks are indeed really very difficult and can't and shouldn't be deliberately over-simplified : that risks fooling people into thinking they immediately understand things when in fact they need years of training. And on occasion this legitimate concern slides into elitist snobbery. Far more common, though, is that we as astronomers just have too much to do to spend their time doing good interface/experience design, though I'm glad to see that there is at least a movement towards encouraging more user-friendly software*.

* One of the stranger objections, which to be honest I couldn't quite get my head around, from the referee was whether or not there was any real evidence that by making things easier to use people would actually use them. I mean, yes, actually yes I do, lots, but the whole question seemed strange to me. Who in their right mind would prefer the more difficult option if the easier one gave results of equal quality ? That bit does seem self-evident to me.

So in that spirit, every operation in FRELLED is accessible via the GUI. You don't need to ever see the code. Every graphical element has an explanatory tooltip. Some things only possible in the previous version by learning some rather hacky tricks from the manual have now been properly and robustly integrated. And every tool panel comes with a simple help menu explaining its individual functions. Once the very basics have been grasped ("here's how to open a file"), most other operations should come naturally. If I can't quite make it so that you can download it and go, I can least make it so that after 5 minutes or less you should have a fair idea of how to do most of the stuff you'll ever need to do.

Which leaves my final point : versatility. There are a lot of good visualisation tools out there, and a lot of good analysis tools too. Much less common are tools that do both. What I've tried to do with FRELLED is seize this unoccupied middle ground and strike a balance. It doesn't do the most advanced visualisation possible but it does have multiple different ways of viewing data : most programs either do 2D or 3D, rarely both, let alone giving a choice of volumetric or isosurface displays. And while it doesn't do every analysis operation possible, it does enough of the basics. At least, for what I need to do. The goal is that users shouldn't have to switch between packages unless they need something which is really quite specialist.

... bugger, I forgot to mention the analysis tools ! I should give a brief rundown, I guess. One of the main things is you can easily click on a source and create an object you can use for cataloguing, masking, and analysing the target. You can give it a human-readable name so you'll remember it and find it again later. This all sounds trivial and honestly it is but nothing else seems to let you catalogue sources quite like this, which to me is still bizarre (though I gather that CARTA has something similar). You can then use those objects for overlaying optical data, adding annotations and axes (both for creating figures but also for training new observers), generating contours plots, and even doing some basic spectral measurements if you also have miriad installed. It does, in short, far from everything, but still quite a bit.

Example figure of some HI contours overlaid on an optical image, created entirely in FRELLED. It sounds minor but the the colour bar on the right was created automatically, which was otherwise a mind-numbingly tedious process that we'd only ever do if we were sure we'd got things just right. Making that automatic allows the user to experiment. The annotation objects and labels were created just by placing the cursor, typing in a toolbox and pressing a button. No need to open PhotoShop ! 

The interface plays a role in versatility too. There's a lot of options for setting things that previous were only visible under the bonnet, i.e. by tinkering with the code itself. Now, for example, you can manually configure the setup to optimise the display for smaller files, or (if you're feeling brave) to go for even larger data sets – all of which include checks to prevent the user from doing something really stupid to the file, or for correcting their mistakes if they insist on being a damn fool about it.

And what I really hope people will do is things I can't predict. I want them to be able to experiment with viewing their data in different ways, rather than sticking with what they already know. I want them to try stuff because it's so easy there's no point in not trying it, without feeling like they're going to muck up they're whole project if something goes wrong. And I want them to be able to load as many different sorts of data sets as they can, not be restricted to just one particular frequency. 

Most of all, I'd like users to feel like they're doing data visualisation (and taking it seriously as a much-needed aspect of analysis) without it feeling like it's a yet another skill set they need to learn. Users should, I hope, be able to learn Blender as astronomers, not become fully-fledged Blender artists who are also interested in science. Ultimately, making fun-looking 3D figures should be normal and for everyone, not a cool-looking bonus option that requires yet more specialist training in an already overloaded discipline.

What's Next ?

Aaaargh ! It happened again ! When I first looked in Blender post-2.79, the testing versions of 2.8 were available that had a version of the EEVEE internal real-time rendering engine that didn't work well. In FRELLED, the cube stays visible at full resolution and a high frame rate. In the pre-release of 2.8 there was a huge, horrible lag in updating the display which would have made it unusable. You can't freely explore a data set like this unless you're a masochist.

Then in the actual releases of 2.8 and later versions this was overcome. And they offer a really powerful extra option : you can adjust the colours of the images after loading them in, which you can't in Blender 2.79. One of the major restrictions of FRELLED is that loading the data sets can be quite slow in comparison to other programs, especially since if you get the colour scaling wrong you can't adjust it. So this alone makes recoding FRELLED, once again, worth doing. It would also give greater performance, allow for larger cubes, have direct viewing in VR, and a bunch of other things.

Recoding FRELLED the next time round won't be the same level of challenge as here, not even close. The Python syntax is more like a modified, incremental change from that of Blender 2.79 with only a few parts which were completely replaced. So this is doable, but it's not happening anytime soon because I did to some actual science again (watch this space). There are, though, a bunch of other features I'd like to add when I eventually get back to this :

• Support for really large cubes (tens of gigabytes) and higher performance for smaller ones
• Automatically overlaying data from online catalogues, to make source identification easier
• Bring back the particle viewer from the old version (I had to drop this just because the project was ballooning into the unsustainable)
• Allowing 2D files, just for the sake of it
• And a whole bunch of under-the-bonnet changes that would make the experience still more natural and faster.

---

So that's this paper. No science here, just a development that's been in progress since 2012. Maybe this, the 13th year, will be the one in which finally people start to use it. Well, you never know.

Meandering in Malta

Yes, another travel post... well gosh, I have been busy, haven't I ?

Yes, yes I have. And I'm very glad to have an entire month of staying put before I need to go anywhere else again. I actually quite like being in different places, I just hate going between them. Especially on this particular trip, where out of the total of 10 days away, three of those had 10-12 hours of travelling apiece. That's a bit much for my liking.

Anyway, at the end of the last post I was just about to embark to Malta. This was an uneventful but boring trip : 6am start for the taxi to the bus stop, two hours on a bus (quiet a scenic route in daylight, lots of barren rocky mountains), two hours in the airport, couple of hours or so flight to Rome, 2.5 hours layover, 1.5 hours or so to Malta, half hour taxi to the apartment. It was indeed a full 12 hours of just moving from place to place interspersed with long bouts of absolute nothingness.

Worth it ? Yeah, I'd say so. First because the conference (and it was a proper conference this time) was useful, but also because Malta is a fascinating place with history in every step. I'd definitely come back.

 

Melting in Malta

Why Malta ? The "Astronomical Data Analysis Software & Systems" conference was recommended to me at last year's All Hands (the same meeting as in Granada this year). Everyone there really seemed to like my code – though predictably I had exactly no feedback on it in the intervening year – and someone suggested ADASS would be a good venue to promote it to a wider audience.

This year's event, the 34th in the series, was hosted by the University of Malta. I'd booked my trip for the day after the All Hands, figuring it would be better to have a spare day somewhere I'd never been before. A wise decision. I should also mention (and I swear nobody's sponsoring me for this) that this included a free taxi from booking.com (with whom I found accommodation) to the apartment. That's undoubtedly the most useful free perk a travel site has ever offered me. Especially since the taxi driver was good enough to give me details for booking the return trip.

Both Granada and Malta were about the same temperature, roughly 10-20 C, slightly warmer but much, much more humid in Malta. The ADASS website says, "the weather in Malta is usually cold and wet at this time of year. Warm jackets and umbrellas are recommended" which is plainly ludicrous. Everyone was dressed for summer, and rightly so because wearing a "warm jacket" would have been like wearing a sign which said, "please hurt me". Umbrellas would be helpful : one day it did nothing but rain, and the rest of the time there were occasional light showers. I managed just fine with my light summer rain jacket. On occasion I even needed a jumper.

I arrived in my apartment at about 7pm, when it was already dark. This turned out to be thoroughly deserving of its 7.3 rating on booking.com. Basic, to be sure, but in an absolutely first-rate location. 10 minutes walk (or less) from the tourist centre with, thank goodness, not a single intimidating street gang in sight (unlike Padova). In fact it was right next to St Elmo's fort. The area itself was largely quiet. Inside it was certainly... standard, and things could do with some plastering and a lick of paint, especially the stairs; the instruction not to wear shoes inside was very much a case of shutting the door after the horse has bolted. But the most important things were all perfectly fine. The bed was comfy, the fridge and microwave worked, the shower was surprisingly good (if small), there was a kettle and... a huge supply of Tetley tea. I'd go there again.

Basic, to be sure, but quite comfortable all the same.

The ceiling beams would however make this unsuitable for anyone taller than 5'10" or so. I'm short enough that this wasn't a problem for me, though I did have to stoop a little in the shower. The air conditioning was very effective though, not needed so much for the temperature as the humidity.

Mind you, I do sympathise with people complaining it wasn't suitable for its advertised maximum of 4 people, though it does clearly specify it has one double bed and one bedroom. Unless you're really, worryingly close with some very good friends, three would be silly, and four would be an absurd joke.

I had that first evening and the next day to explore my surroundings. One of the first things I was struck by was just how very, very British Malta feels. This was my first time in which I got to use my UK adaptor outside of the UK. But there are a lot more vestiges of British rule all over the place : phone boxes, post boxes, pedestrian crossing buttons, shops (Accessorize, M&S, Spar – among others), even a Cardiff Snack Bar (!)  that I saw from the taxi on the way back. Tourists too seem to be predominantly British. Even the scattered rain showers feel familiar.

From the Spar I bought Ribena, and felt like I'd come home.

Yet in every other respect it's blindingly obvious that this is worlds away from Britain. The streets are lined with palm trees and cactus. The November sun is still powerful enough that anything more than a T-shirt is usually unnecessary, sometimes even silly. Maltese is heard everywhere. And of course the architecture is altogether different. I have to say that many streets look run-down, almost dilapidated, but they are very much actively used and lived-in; a little bit of a culture-shock there. The posher places are very much cheek-by-jowl with those operating on a lower budget, and from the exterior they're not always easy to tell apart.

By the standards of Valletta this is a very wide and well-kept street.

My main bit of non-science excursion was to indulge my favourite tourist pastime of wandering aimlessly. Around Valletta this is an excellent strategy. As a capital city this is absolutely miniscule but very dense, with history in every step, and you can't fail to find something worth viewing everywhere you go. Doing a pre-planned route would be both boring and unnecessary. 

The narrow streets make finding somewhere to take a good overview photo difficult. This was about as good as I could manage. They need a few tall towers like Bologna !

The maximum elevation may only be 250m or so but the streets are very hilly. You get a lot of exercise walking around Malta.

The weather was... sometimes like this, sometimes bright sunshine. But I wouldn't want to come back in summer when it's 40 C.

The only "planned" activity I did was St Elmo's Fort, which was right next to my apartment so I could hardly avoid it. In fact I went twice. The first time I spent a good couple of hours walking around the whole fort, reading the signs and going through the series of really very good exhibits which cover the entire history of the island : from prehistory and classical antiquity to the Great Siege (1565) and WWII. I can summarise it all very simply. Malta has seen some serious shit. At one point it was reckoned among the most bombed places on Earth. But it's still here, the hulking fortresses of the knight's era still indomitable edifices of stone with walls that must be 20m thick. It is, in short, well worth visiting.

What exactly this is, if it's even part of the fort itself or technically outside it, I'm not sure.

The second time was the day after. As it happened the 10th November was the day the fort was opened for free with a military parade by the In Guardia re-enactment society, who do a parade of the knights of St John (being also Remembrance Sunday, this was preceded by a short memorial service of five soldiers which is basically identical to British remembrance practises). For a tiny local group they did an excellent job, and the acoustics of the fort made the muskets sound more like cannons. Incidentally, I also saw the 4pm ceremonial firing of a cannon from the Saluting Battery, so I have a good point of comparison.

The battery is about 15 minutes from the fort. You can pay a tiny fee to be on the platform when they fire the cannon, but I watched from the balcony like a cheapskate.

The Conferencing

This was my first ADASS so I didn't really know what to expect. Nobody made the obvious BADASS pun; perhaps that joke has long since run its course. I knew only three or four other people present but this does force one to mingle... which began almost instantly. The first day was a half-day session of tutorials. The first I simply watched along with, but the second was on user experience (UX). More on this in a moment, but the tutorial divided us all into random groups of six with the task being to design an app. The goal of this was not to make something functional but just to experience the design process as guided by a professional. 

A professional designer in astronomy ? Indeed. The ASTRON group in the Netherlands make so much software that they've hired a full-time UX designer who ran this course. The instructions filled me with dread : create a persona for people who will use your tool, write a story about how they'd use it... lordy, that sounds like corporate nonsense. Luckily for me, our group alone decided to have some fun with this. We decided to make an app to alter users of potential meteorite strikes. Our user persona was called Harald and he had a beloved cat (Whiskers) and a herd of cows. He was paranoid about meteorite strikes and his favourite activity was "not being hit on the head by meteorites". We even produced a visual of what the app ("Don't Look Up*") would look like, with a little warning message about the likely chance of a strike and options to get more details.

* Tagline, "Look At The App". 

Now, all of this is something one could do oneself without any guidance whatever. What I found the process did, however, was massively speed everything up. By delegating roles (though not very strictly) to each of the team members, everyone thinks through the process according to different perspectives. Consequently the creative juices really start flowing very quickly. How much warning does Harald need ? How does he want to receive alerts ? Does he want it to run in the background ? Do we need an Enterprise Edition that could provide warnings for multiple sites ? And so on.

The rest of the conference followed a more traditional structure. Normally conferences are subject-orientated, but this one is selected based on software. This is interesting because it brings together a much wider pool of expertise, with many people from software backgrounds who aren't professional astronomers at all. It makes the tea breaks an interesting experience, but the talks also are a lucky dip. Some were incredibly technical, more like computer science than astronomy*, others were about the history of project development, while a good chunk were on topics regular science conferences would normally consider too "soft".

* To the poor sod giving the talk about field gate programmable arrays, man, I have no idea what you were talking about, but bloody hell it sounded intelligent.

I'm going to limit myself here to the latter. There was a great deal of discussion about UX, which is about more than the software interface and actually even higher-level than that : how do users really use the software ? As various presenters said, traditionally astronomers aren't good at that. We write code that gets the right answer, but we don't do the (considerably easier) task of bothering to provide them a useful, intuitive interface. And we really should, because a bad interface means mistakes and the very inaccuracies we wanted to avoid ! Everyone likes the technical aspects and tries to avoid the social. But we shouldn't : these soft aspects can have hard results.

I have to also add that there remains an attitude in some quarters that our codes should be difficult to use, that they should only be accessible to an elite. This was firmly rejected here, with the overwhelming mood being to encourage everyone to actually think about interface design and take it seriously. No, it's not magnetohydronamics, but it's still important. I particularly liked the argument that we should presume users have the ability to understand the tool and it's only bad design which prevents them : the analogy made was that wheelchair users are only "disabled" when you don't provide them with lifts. 

As for my talk, I'm pleased to say that everyone really liked it. "I wish I'd thought of that", "I never would have thought of doing it that way" and even "Brilliant !" and "I wish I had that when I did cell biology" were mentioned. I was pleased as punch. There was even a question from the organisers afterwards as to whether we could organise ADASS in Prague. Well, I hope so. Next year it's in Germany which is much easier to get to than Malta so I'll probably try and go along; it was a welcoming affair and an interesting experience. In any case, it's clear that I need to keep promoting my code and that yes, other people want this to exist.

It was quite a large conference, with ~270 in-person attendees and another 60 or so online.

I'll close the science section with a sting in the tail. There were several nice history talks, including one on a 20-year project to digitise the entire collection of 500,000 photographic plates in Harvard. But there was also a warning. The incredibly popular DS9 software, which gets ~40,000 downloads per year, is in serious danger of running out of funding. This to me is almost literally unbelievable; it's such an indispensable tool that I just don't understand how any manager could be so ludicrously blind as to not see how its ~1 FTE requirement hasn't given back that investment a thousandfold and more over the years. Cancelling DS9 ? Have you gone completely mad ?

Ah well, a warning to us all. Anyway, the only conference socials were the welcome reception in which I managed to survive a ferocious onslaught of free wine, and a walking tour of Valletta. This helped put things in a bit more context. The guide was good, but a bit more "I've done this ten thousand times before" than the Alhambra guide. I asked her what the Maltese thought of the British when they decided for independence : was it a case of "get the hell out ?" or "we don't hate you, but please just leave". She said the latter. Which is as good as a colonial power can hope for, really.

I really liked the Tritons fountain. The way the water cascades in a shower from underneath the bowl, rather than just flowing over the edge, was somehow quite fascinating.

She also had quite an unusual take on the Great Siege. In Fort St Elmo this is portrayed as it usually is, as a heroic victory against the odds, a huge turning-point in the war between the Ottoman Turks and European Christendom. The guide, however, thought it wasn't anything in particular to celebrate, but more just pure luck : the Turks left because they were tired and wanted to go home and were worried by rumours of Christian reinforcements, rather than actually being beaten by the knights. Yet the exhibits at the fort portrayed this as a result of clever trickery in which the rumours were deliberately engineered by the knights.

Personally, for whatever it's worth, I stick with the classical view. Look, if you can hold out for so long against your enemy that they refuse to fight you anymore, if they'd rather turn tail instead of offering battle, then you've won.  It doesn't matter if this is because you actually beat them to a bloody pulp or convinced them through trickery that you could beat them to a pulp if they were foolish enough not to run away.  How you achieve victory has no bearing whatever on whether you have achieved victory. In that respect, the only thing that matter is that the enemy stop fighting. And I doubt there's a general in the world who'd disagree. 

And besides, for the massively-outnumbered knights to take out a third of the invasion force for the cost of one single fort is nothing to be sniffed at by anyone's standards.

All this being done, I went home. More conversations followed at the airport with other conference escapees who were, I'm delighted to say, enthusiastic about my presentation. Which is encouraging. Even DS9 took years to get any attention, but now practically everyone uses it : as an easy-to-use tool for quick inspection, it's second to none, and does a lot more besides. Of course the warning is that even that level of superlative success doesn't necessarily bring any benefits with it.

The return trip was just as long and uneventful as all the rest. Malta airport security have an everything scanner so that took all of five minutes. It's annoyingly busy inside (not much bigger than Cardiff despite being a popular tourist destination) but not awful. Zurich (shown below), where I had a four hour layover, is really very good indeed : absolutely massive, but with plenty of places to sit and charging points galore. Overpriced to be sure (I spent 18 EUR on Burger King because it was the cheapest option) but at least you can easily find a quiet place to sit. And so, feeling like I'd been gone for a full month rather than a mere ten days, back I went to Prague.