home whoami phone maas rPI4 FreeBSD

more jq tricks

install ~ configure ~ DHCP ~ commission ~ deploy ~ jq-tricks ~ ssh-in ~ jq-tricks++ ~ nagios


I previously covered basic usage of jq with MAAS a couple of posts ago. Now I want to dig in a little bit, and see what else I can do with it.

sorting natively with jq

In my previous jq post, I showed how to use Linux command line utilities to sort lines in a multi-record jq output. It's not the only way -- nor maybe even the easiest way. The jq tool has a native sort capability. Let's take a look.

Suppose we have a random list of machines that I quickly created on one of my local MAAS installs -- haven't even bothered to commission them, since that doesn't matter here. If I run the command:

maas admin machines read | jq -r '(["HOSTNAME","SYSID",
"TAGS", "POOL","ZONE"] | (., map(length*"-"))),
(.[] | [.hostname, .system_id, .tag_names[0] // "-", 
.pool.name,.zone.name]) | @tsv' | column -t

on my very casually defined set of new machines, I'll get a listing something like this:

HOSTNAME       SYSID   TAGS      POOL      ZONE
--------       -----   ----      ----      ----
nearby-louse   ed8pda  plover    swimming  defense
novel-cod      4ds3eh  dunnet    swimming  fault
nearby-koala   k6cagg  xyzzy     car       twilight
superb-koi     6sqads  xyzzy     swimming  twilight
amazed-hermit  mehgsk  xyzzy     swimming  twilight
daring-minnow  ceamb4  plover    car       demilitarized
ace-colt       hfm84c  zork      tidal     defense
unique-donkey  tsf8tk  plover    tidal     demilitarized
mutual-cod     sxbkyp  bedquilt  tidal     defense
fun-ibex       8pq6qx  zork      car       fault

These aren't really sorted by anything, in particular, so let's just start at the front and work our way in, shall we?

sorting by one parameter

Rather than passing our output through an awk script, we can just use the jq construct for sorting, which is:

sort_by(<fieldname>)

For example, if we wanted to sort by HOSTNAME, we could try:

maas admin machines read | jq -r '(["HOSTNAME","SYSID",
"TAGS", "POOL","ZONE"] | (., map(length*"-"))),
(.[] | [.hostname, .system_id, .tag_names[0] // "-", 
.pool.name,.zone.name] | sort_by(.hostname)) | @tsv' | column -t

This doesn't give us anything like what we want, just a mysterious error:

jq: error (at <stdin>:2911): Cannot index string with string "hostname"
HOSTNAME  SYSID  TAGS  POOL  ZONE
--------  -----  ----  ----  ----

Huh? Don't you just link pipes together, and they work?

jq is not UNIX (jqNU?)

No, that's UNIX, but that's not necessarily jq. Too see what's wrong, we have to break down the earlier jq statement a little more -- you know, the one that just gives us the unsorted list:

maas admin machines read | jq -r '(["HOSTNAME","SYSID",
"TAGS", "POOL","ZONE"] | (., map(length*"-"))),
(.[] | [.hostname, .system_id, .tag_names[0] // "-", 
.pool.name,.zone.name]) | @tsv' | column -t

If we read it from left to right, we have the MAAS command that generates the JSON to begin with:

maas admin machines read

We pipe that through jq, with raw output:

| jq -r

We give jq some series of commands, between the single quotes:

'(["HOSTNAME", blah, blah, blah | @tsv'

Finally, we pipe that to column to make the columns line up nicely:

| column -t

So essentially, from a UNIX perspective, we have a data source and two filters:

maas <options> | jq <options> | column <options>

The options we give to jq most definitely don't work like UNIX.

where's the data coming from?

A great question to ask yourself when building jq blocks is, "Where's the data coming from?" Let's take a closer look at the jq options:

'(["HOSTNAME","SYSID","TAG","POOL","ZONE"] |
(.,map(length*"-"))),
(.[]  |
[.hostname, .system_id, .tag_names[0] // "-",.pool.name,.zone.name]) |
@tsv'

Ask yourself where the array data is coming from. Does anything in this line retrieve data?

'(["HOSTNAME","SYSID","TAG","POOL","ZONE"] |

Nope. Those are literals, which don't retrieve any data at all. How about the next line?

(.,map(length*"-"))),

No, that line acts on data it gets, but it doesn't get data. And the next one?

(.[] |

Yep, that's it. The .[] command tells jq to iterate over what it gets. The rest of the line just tells jq which of the fields in the .[] iteration to actually retrieve.

So by process of elimination, there's only one place we could sort data -- where it comes into the command, that is, where we currently iterate over the incoming JSON arrays, .[].

so how does "sortby()" work?

The "sortby()" subcommand operates on whatever it gets. In this case, we're sending it a JSON array of machine datasets. So there are two things wrong with our first attempt:

maas admin machines read | jq -r '(["HOSTNAME","SYSID",
"TAGS", "POOL","ZONE"] | (., map(length*"-"))),
(.[] | [.hostname, .system_id, .tag_names[0] // "-", 
.pool.name,.zone.name] | sort_by(.hostname)) | @tsv' | column -t

First, it's set up to handle a single JSON dataset, and sort it by a single parameter -- sort_by(.hostname) -- so there's actually nothing to sort, hence an error. Second, we trying to pipe a formatted entry from an array of JSON datasets through it, which doesn't even make any sense: you can't sort one thing.

To make the command work, we have to make sort_by itself the data source. In other words, we want jq to sort the record stream before it gets our field list, like this:

maas admin machines read | jq -r '(["HOSTNAME","SYSID","TAG","POOL","ZONE"] | (., map(length*"-"))),(sort_by(.hostname)[] | [.hostname,.system_id,.tag_names[0],.pool.name,.zone.name]) | @tsv' | column -t

As we can see, this command works as we expect:

HOSTNAME       SYSID   TAG       POOL      ZONE
--------       -----   ---       ----      ----
ace-colt       hfm84c  zork      tidal     defense
amazed-hermit  mehgsk  xyzzy     swimming  twilight
daring-minnow  ceamb4  plover    car       demilitarized
fun-ibex       8pq6qx  zork      car       fault
mutual-cod     sxbkyp  bedquilt  tidal     defense
nearby-koala   k6cagg  xyzzy     car       twilight
nearby-louse   ed8pda  plover    swimming  defense
novel-cod      4ds3eh  dunnet    swimming  fault
superb-koi     6sqads  xyzzy     swimming  twilight
unique-donkey  tsf8tk  plover    tidal     demilitarized

Reading the new command from left to right helps understand it. Basically, it reads: "Take the data for all machines from MAAS, display a heading line and a horizontal rule line; then sort the machine data by hostname, and show the following 5 parameters, in tab-separated rows, lined up in neat columns."

checking our work

We can check our work by trying different sorts:

maas admin machines read | jq -r '(["HOSTNAME","SYSID","TAG","POOL","ZONE"] | (., map(length*"-"))),(sort_by(.system_id)[] | [.hostname,.system_id,.tag_names[0],.pool.name,.zone.name]) | @tsv' | column -t

This command sorts by system ID:

HOSTNAME       SYSID   TAG       POOL      ZONE
--------       -----   ---       ----      ----
novel-cod      4ds3eh  dunnet    swimming  fault
superb-koi     6sqads  xyzzy     swimming  twilight
fun-ibex       8pq6qx  zork      car       fault
daring-minnow  ceamb4  plover    car       demilitarized
nearby-louse   ed8pda  plover    swimming  defense
ace-colt       hfm84c  zork      tidal     defense
nearby-koala   k6cagg  xyzzy     car       twilight
amazed-hermit  mehgsk  xyzzy     swimming  twilight
mutual-cod     sxbkyp  bedquilt  tidal     defense
unique-donkey  tsf8tk  plover    tidal     demilitarized

We can also try pool name:

maas admin machines read | jq -r '(["HOSTNAME","SYSID","TAG","POOL","ZONE"] | (., map(length*"-"))),(sort_by(.pool.name)[] | [.hostname,.system_id,.tag_names[0],.pool.name,.zone.name]) | @tsv' | column -t

which also works as we'd expect:

HOSTNAME       SYSID   TAG       POOL      ZONE
--------       -----   ---       ----      ----
nearby-koala   k6cagg  xyzzy     car       twilight
daring-minnow  ceamb4  plover    car       demilitarized
fun-ibex       8pq6qx  zork      car       fault
nearby-louse   ed8pda  plover    swimming  defense
novel-cod      4ds3eh  dunnet    swimming  fault
superb-koi     6sqads  xyzzy     swimming  twilight
amazed-hermit  mehgsk  xyzzy     swimming  twilight
ace-colt       hfm84c  zork      tidal     defense
unique-donkey  tsf8tk  plover    tidal     demilitarized
mutual-cod     sxbkyp  bedquilt  tidal     defense

sorting by more than one parameter

To sort by more than one parameter, simply include a comma-separated list of parameters for the sort, in order. For example, to sort by tag first and pool next, use this incantation:

maas admin machines read | jq -r '(["HOSTNAME","SYSID","TAG","POOL","ZONE"] | (., map(length*"-"))),(sort_by(.tag_names[0],.pool.name)[] | [.hostname,.system_id,.tag_names[0],.pool.name,.zone.name]) | @tsv' | column -t

This will give you:

mutual-cod     sxbkyp  bedquilt  tidal     defense
novel-cod      4ds3eh  dunnet    swimming  fault
daring-minnow  ceamb4  plover    car       demilitarized
nearby-louse   ed8pda  plover    swimming  defense
unique-donkey  tsf8tk  plover    tidal     demilitarized
nearby-koala   k6cagg  xyzzy     car       twilight
superb-koi     6sqads  xyzzy     swimming  twilight
amazed-hermit  mehgsk  xyzzy     swimming  twilight
fun-ibex       8pq6qx  zork      car       fault
ace-colt       hfm84c  zork      tidal     defense

You can even try three parameters. Note that I set this up with the two hosts "superb-koi" and "amazed-hermit" -- so we can use a command like this to see how multiple parameters work:

maas admin machines read | jq -r '(["HOSTNAME","SYSID","TAG","POOL","ZONE"] | (., map(length*"-"))),(sort_by(.tag_names[0],.pool.name,.hostname)[] | [.hostname,.system_id,.tag_names[0],.pool.name,.zone.name]) | @tsv' | column -t

Note that "superb-koi" and "amazed-hermit" have changed places in the listing below:

HOSTNAME       SYSID   TAG       POOL      ZONE
--------       -----   ---       ----      ----
mutual-cod     sxbkyp  bedquilt  tidal     defense
novel-cod      4ds3eh  dunnet    swimming  fault
daring-minnow  ceamb4  plover    car       demilitarized
nearby-louse   ed8pda  plover    swimming  defense
unique-donkey  tsf8tk  plover    tidal     demilitarized
nearby-koala   k6cagg  xyzzy     car       twilight
amazed-hermit  mehgsk  xyzzy     swimming  twilight
superb-koi     6sqads  xyzzy     swimming  twilight
fun-ibex       8pq6qx  zork      car       fault
ace-colt       hfm84c  zork      tidal     defense

summary

So there's a little more jq for you, but still a lot more to learn. Might as well keep going with this, it seems to be useful to some folks -- so more jq to come.


Updated 2021-02-11 Thu 18:01 by stormrider (stormrider)


Copyright (C) 2020 by Bill Wear. All rights reserved, but asking to use is permitted and welcome.