We’re using this at the moment. This is very crude in the sense it’s a mini‑ETL language for a pipeline that’s based around tabular data, particularly.

If you do quite a bit of ETL, one of your pain points is normally, as they get complicated, "What is the agreement between different stages of the pipeline?"

For example, we have this thing at the moment we’re calling — that’s about to be a post, I think — data package pipelines. Basically, ETL pipelines run around packaged input of each stage of the data package. They output the next stage of the data package.

Rather than put it into a structured database, one of the things I like at the moment is you could just use flat files. If you have a DDL and you’re dumping data packages, if you’re getting a lot of raw log data in, you could actually run a lot ...

That’s really great. One of the things I think about is that certainly when you’re having large ingestion of raw data, obviously one of the other things you can do is start... often, at the beginning, you could just be dumping...

Yeah.

No. I haven’t.

You enrich it, basically. There’s an enrichment.

Those are things that are valuable. We can stop on that one, then, for the moment. I’d like if you could tell me a little bit about this...

We did, exactly. I’m just saying, you might want to publish to that, and then convert to JSON for your JSON API.

As an experience, maybe you want to have a data package intermediary there. They’re going to publish from maybe Excel.

One is common descriptive metadata. You could adopt some of the similar descriptive metadata. The other is this question on publisher/user experience. For example, you might want to have a JSON API. At the end, that’s what you’re defining in Swagger.

There’s the thing. I’ve got you. OData, by the way, is like, OData and data packages have quite a lot in common. OData is quite similar. I want to stop there on what I’m saying about data packages. What I would offer is terms of value whatever route you’re going ...

It doesn’t want nesting. Why? It’s because it’s hard to do atomic updates of individual items. it becomes a very complex thing to track. Let’s say, for example, you retrieve a post from the server. If that’s nested under the user, then you have to go to...

To un‑normalize it, to un‑nest it. If you read the Redux docs, they’ll talk quite a bit about why you need to renormalize your data.

One of the funny things about Redux is that if you notice the design of a Redux store, your Redux store is like a local cache of your remote database. One of the things you do in Redux is they encourage you to normalize your data.

Like front end. Do people use React and Redux?

Like React?

Right, or it’s a JSON field, but is it a good example? Who here knows Redux?

If you have foreign keys, you can even do complex joins. You can convert JSON back into a normalized structure with tables with no nesting.

Right, but you’ve got to ask yourself, is most of your data designed for consumption by web applications, or is it for consumption back into a database, or back into something else? I’m only pointing this out. I’m not saying you can convert CSV to JSON incredibly easily, like it’s ...

For your exchange with a lot of your data, once you’ve got a JSON field, for a lot of the tools, you want to get data back into, it’s not especially convenient, unless you’re building for web application, unless your web APIs are optimized for consumption by web applications.

You can express however you want it. We could even have GraphQL etc. In some sense, those are optimized for a certain kind of consumption. GraphQL is a JSON API optimized for consumption of mobile or other devices, where bandwidth matters, or complexity of queries matter.

It was worse as a structured publishing format, but it was easier to consume. It won. One of my comments is that JSON is great, and you can convert to it, but for example, it’s great for APIs for geeks.

Even HTML4 is better than XHTML.

From a point of view of publishers, HTML is pretty rubbish. HTML has no semantic structure, basically, until HTML5.

Because the people who write the spec are often involved in producing the data, they tend to focus on their concerns as publishers to express what they want to express. Actually, the success of many of your formats is going to depend on consumption.

Tim Davies is the guy who was quite involved in IATI. The thing is, in life, how do I put it? Many formats get dominated by publisher concerns, not by consumer concerns.

All I’m pointing out is that, for example, I’ve had this debate with standardization efforts. For example, this thing called the Open Contracting Partnership.

Yeah, but in that case, you’re basically just doing NoSQL in a relational database. You can do it.

Once you start doing that, you have all kinds of fun normalization games. It’s not trivial to take JSON back into a relational database.

Now, when you have JSON, people do all kinds of fun stuff with JSON. For example, they nest entities. You take one entity, and then inside of it, you embed something else. For example, you might have users, and then you put their posts inside them.

Just take an example. One of the standard things people do with data is put it in a relational database, or put it in spreadsheets. They dominate almost any other system — even NoSQL — by orders of magnitude today in terms of installed base.

The next step has to be, in all of this, it’s just like, "Here is all the tooling you can do. What platform integration can you build?" Going back just a point about see JSON, JSON’s awesome, you can all kinds of stuff with your JSON.

The question is, then, on the tools, what we want to remember at the very beginning is I should have one click to install, just like you can type npm install, and the library installs. Now, geeks do that, but whether you’re in R, or whether you’re in Excel, or ...

Here, what you also have, really is, if you know what I mean by OLAP, you probably want to get into dimensional, not just saying, "Hey, I’ve got this columns," but columns go together. For example, these three columns together describe a customer. These three columns together describe a department, ...

People know what I mean by a pallet. By the way, containers don’t all look the same, it turns out. There’s ones specialized for particular functions. Then you could say, "Mine is a steel box for pallets that hold bicycles."

Really, by the way, the next level that you probably want if you’re going up, is you’ve got a basic container, like a steel box. Then you say, "My steel box is designed for pallets."

Yes, that’s how you described it.

This, it basically says, "Be in CSV." Basically, it says, "It must be a tabular data package." Then it says what extra columns or stuff you must have.

There you go, the fiscal data package. This, what it does is it says, "It’s a data package. It’s a tabular data package." It’s got to have tables in it, CSV. Then it says what columns you’ve got to have. It has a model for that.

You know that. You can do that kind of integration. Here, the other thing I was going to mention, actually, is fiscal data package. There’s a whole specification that goes beyond just having a tabular data package. I think you search for fiscal data package.

You could do things like install it in Travis. Every time you push a file to GitHub, it will check whether your data is valid, and fail or succeed on your pull request.

This is an error reporting system that’s built for JSON table schema. You can do, if you want, cool things like once you have this, and this is a Python library, and there’s also a JavaScript library. Now, we’re implementing libraries in every language.

Right. The thing is, all I’m saying is, this tooling, that’s the really valuable part of format. I can make up format, whatever, building this kind of tooling, so this tooling is built for data JSON table schema.

From the revolution.

There’s also weird stuff, like Excel may show you one date format, but when it exports, it exports in a different date format. It’s things like that that turn out to be hard to do...

It turns out that just explaining good date formats is harder than you think. It’s not just explaining the date format. For example, many of them go, "Oh, OK, but how do I go to Excel and change the date format?" They don’t know how to change date formats in ...

Most ordinary users don’t even know what column number one is. You want to draw it like they visualize it. That’s why we try to start drawing out, but it’s harder than you think. [laughs] For example, how do you explain date formats?

Originally, we could just say something like, "Header column number one is empty." This is a geek message. This was written by the...