endianness assumptions (perhaps in code, perhaps in tests)

[llimeht]

Version: 0.12.2

The following tests currently fail on the s390x architecture which distinguishes itself as probably the only big endian architecture that the tests are run against:

131s test/sasdataloader/utest_sasdataload.py::test_load_file[test_case10] FAILED [ 84%]
131s test/sasdataloader/utest_sasdataload.py::test_load_file[test_case11] FAILED [ 84%]
132s test/sasdataloader/utest_sasdataload.py::test_load_file[test_case12] FAILED [ 84%]
132s test/sasdataloader/utest_sasdataload.py::test_load_file[test_case13] FAILED [ 84%]
133s test/sasdataloader/utest_sasdataload.py::test_load_file[test_case14] FAILED [ 84%]
133s test/sasdataloader/utest_sasdataload.py::test_load_file[test_case15] FAILED [ 84%]
133s test/sasdataloader/utest_sasdataload.py::test_load_file[test_case16] FAILED [ 84%]
133s test/sasdataloader/utest_sasdataload.py::test_load_file[test_case17] FAILED [ 84%]
133s test/sasdataloader/utest_sasdataload.py::test_load_file[test_case18] FAILED [ 84%]
133s test/sasdataloader/utest_sasdataload.py::test_load_file[test_case19] FAILED [ 84%]
133s test/sasdataloader/utest_sasdataload.py::test_load_file[test_case20] FAILED [ 84%]
133s test/sasdataloader/utest_sasdataload.py::test_load_file[test_case21] FAILED [ 84%]
133s test/sasdataloader/utest_sasdataload.py::test_load_file[test_case22] FAILED [ 85%]

These are all the XmlTestCase tests for test_file_load.

It's not clear to me at this stage whether this is the fault of the xml reader that is incorrectly reading the data, the fault of the code reading the comparison data, or the fault of the comparison test itself. (I have a feeling it's the last of those but I don't know what is in these base64-encoded blobs)

pytest does automagically diff the items for us but it's not that readable as you can imagine. The only difference from test_case10 is:

135s E             Differing items:
135s E             {'79680main_1D_2.2_10.0': {'name': '79680main_1D_2.2_10.0', 'data_contents': {'I': {'value': {'type': 'ndarray', 'enco...dowfactor', 'Qmean', 'dQl', 'dQw'], 'expected_orders': [['Q', 'I', 'dI'], ['Q', 'dQ', 'I', 'dI']]}, 'mask': None, ...}} != {'79680main_1D_2.2_10.0': {'name': '79680main_1D_2.2_10.0', 'data_contents': {'Q': {'value': {'type': 'ndarray', 'enco...dowfactor', 'Qmean', 'dQl', 'dQw'], 'expected_orders': [['Q', 'I', 'dI'], ['Q', 'dQ', 'I', 'dI']]}, 'mask': None, ...}}
135s E             
135s E             Full diff:
135s E               {
135s E                   '79680main_1D_2.2_10.0': {
135s E                       'name': '79680main_1D_2.2_10.0',
135s E                       'data_contents': {
135s E             +             'I': {
135s E             +                 'value': {
135s E             +                     'type': 'ndarray',
135s E             +                     'encoding': 'base64',
135s E             +                     'contents': 'QFVVVMmF8G9AUiKMFUyYX0BPrdl/YrauQEyML4N7SiNASYHHeaa1C0BHPL+xW1c/QETcC3gDRtxAQrXcxj8UEkBA3hsImgJ1QD6beANG3F1AO7scQyylekA5QgW8AaNuQDbFbVz6rNpANIhE0BOpKkAzEwvg3tKJQDFIf8uSOipAL3rULDye7UAtHp4bCJoCQCsuYx+KCQNAKVdl/YraukAnDKV6eGwiQCVq9PDYRNBAJEmF8G9pREAi/Sncclw+QCGdkWhysCFAIKrVe8f3e0AfKpFTefqYQB0JPIn0CihAHDEKE384xUAaIeeWfK6nQBj7L+xW1dBAF8Qd0aIeo0AW1SflIVdpQBT6W5Yoy9FAFFTmW+oLokAThoCdSVGDQBJwC8vmHQBAEd0+1SflIUARTIYFaB7NQBCQ1aW5Yo1ADuuaF23az0AN8Rcu8K5TQA12ZRbbDdhAC8JgLJCBw0AK+sHSnccmQAoqebutwJhACRDqnm7rcEAIHGff4yoGQAcCnxaxHG1ABs2uPmxMWUAF0uxrzoU0QATlf7aZhKFABLYraufVZ0ADrdBSk0rLQAMEkSmIj4ZAArrRjSXt0EABvVZs9B8hQAEC/XXiBGxAAOavzOHFgkAAmhIvrWy1QABIqbz9S/E//srVe8f3ez/+id8Rcu8LP/6ZrpJPIn0//N1jiGWUrz/7i+tbLU1AP/rAeJYT0xw/+iAH3UQTVT/6MaS9ugpSP/lhE0BOpKk/+Pq3VkMCtD/39lVcUucuP/icy31BdD8/95QcghbGFT/2fj0cwQDnP/Z6guh9LHw/9YrQPZqVQj/1KHfuTibVP/Vr7fpD/l0/9Ly+Yc/+sD/049HMEA5rP/K+V1Oj7AM/8vhsImgJ1T/zB/I8yN4rP/K6kqMFUyY/8V8Rcu8K5T/w/keZG8VYP/D9a2WpqAU/8G+yquKXOT/v2Gyon8baP/BpOerdWQw/72WdEsrd3z/wSv9tMwlCP+76sQumJnA/7hnanJkoWz/w7ALy+YdAP+5lWfbsWwg/7Ijv/io86j/sCOFQEZBLP+3TxXnyNGU/7bArQPZqVT/sDdgv114gP+nkWhysCDE/7h5TqB3A2z/uRxDLKV6eP+uEGJN0vGo/7rRKHwgDBD/obFCLMs6JP+kii7Ciyps/6jNIK+i8Fz/ltLlFMIu5P+hdu5z5oGo/6Dn3cpLEkz/jOGj9GZuyP+XwuM+/xlQ/6ZLuhK15Sj/nmdAgPmPpP+bs3yZrpJQ/5w2fkFOfuj/jVHOKO1fFP+TFNL127nQ/5mwPiDM/yD/ixvm5lOGkP+iBvrGBFuw/6obn5i3G4z/julwcYIjXP+DVFQVKwpw/4GicoYvWYj/jVYQrc0tRP+ej9n9Nvfk/3Z6QeV9nbz/eouwosqaxP+bIkJKJ2uA/3kqc3EQ5FT/hLZrYXfqHP+UkE9t/Fzc/2itzS1E3KgAAAAAAAAAA',
135s E             +                     'dtype': '>f8',
135s E             +                     'shape': [
135s E             +                         138,
135s E             +                     ],
135s E             +                 },
135s E             +                 'units': 'cm^-1',
135s E             +             },
135s E                           'Q': {
135s E                               'value': {
135s E                                   'type': 'ndarray',
135s E                                   'encoding': 'base64',
135s E             -                     'contents': 'O99PjZdugj+6SQwCK4eGPzm0yHa+n4o/uB6F61G4jj+cxCCwcmiRP9v5fmq8dJM/Gy/dJAaBlT9aZDvfT42XP5qZmZmZmZk/2c73U+Olmz8ZBFYOLbKdP1g5tMh2vp8/TDeJQWDloD/sUbgeheuhP4ts5/up8aI/K4cW2c73oz/LoUW28/2kP2q8dJMYBKY/CtejcD0Kpz+q8dJNYhCoP0oMAiuHFqk/6SYxCKwcqj+JQWDl0CKrPylcj8L1KKw/yXa+nxovrT9oke18PzWuPwisHFpkO68/VOOlm8QgsD+kcD0K16OwP/T91HjpJrE/RIts5/upsT+TGARWDi2yP+Olm8QgsLI/MzMzMzMzsz+DwMqhRbazP9NNYhBYObQ/I9v5fmq8tD9zaJHtfD+1P8P1KFyPwrU/EoPAyqFFtj9iEFg5tMi2P7Kd76fGS7c/AiuHFtnOtz9SuB6F61G4P6JFtvP91Lg/8tJNYhBYuT9CYOXQItu5P5HtfD81Xro/4XoUrkfhuj8xCKwcWmS7P4GVQ4ts57s/0SLb+X5qvD8hsHJoke28P3E9CtejcL0/wcqhRbbzvT8QWDm0yHa+P2Dl0CLb+b4/sHJoke18vz8AAAAAAADAP6jGSzeJQcA/UI2XbhKDwD/4U+Olm8TAP6AaL90kBsE/SOF6FK5HwT/wp8ZLN4nBP5huEoPAysE/PzVeukkMwj/n+6nx0k3CP4/C9Shcj8I/N4lBYOXQwj/fT42XbhLDP4cW2c73U8M/L90kBoGVwz/Xo3A9CtfDP39qvHSTGMQ/JzEIrBxaxD/P91PjpZvEP3e+nxov3cQ/H4XrUbgexT/HSzeJQWDFP28Sg8DKocU/F9nO91PjxT++nxov3STGP2ZmZmZmZsY/Di2yne+nxj+28/3UeOnGP166SQwCK8c/BoGVQ4tsxz+uR+F6FK7HP1YOLbKd78c//tR46SYxyD+mm8QgsHLIP05iEFg5tMg/9ihcj8L1yD+e76fGSzfJP0a28/3UeMk/7nw/NV66yT+WQ4ts5/vJPz0K16NwPco/5dAi2/l+yj+Nl24Sg8DKPzVeukkMAss/3SQGgZVDyz+F61G4HoXLPy2yne+nxss/1XjpJjEIzD99PzVeuknMPyUGgZVDi8w/zczMzMzMzD91kxgEVg7NPx1aZDvfT80/xSCwcmiRzT9t5/up8dLNPxSuR+F6FM4/vHSTGARWzj9kO99PjZfOPwwCK4cW2c4/tMh2vp8azz9cj8L1KFzPPwRWDi2ync8/rBxaZDvfzz+q8dJNYhDQP/7UeOkmMdA/UrgehetR0D+mm8QgsHLQP/p+arx0k9A/TmIQWDm00D+iRbbz/dTQP/YoXI/C9dA/SgwCK4cW0T+e76fGSzfRP/LSTWIQWNE/Rrbz/dR40T+amZmZmZnRP+58PzVeutE/QmDl0CLb0T+WQ4ts5/vRP+kmMQisHNI/',
135s E             +                     'contents': 'P4Jul41P3zs/hocrAgxJuj+Kn752yLQ5P464UeuFHrg/kWhysCDEnD+TdLxqfvnbP5WBBiTdLxs/l41P3ztkWj+ZmZmZmZmaP5ul41P3ztk/nbItDlYEGT+fvnbItDlYP6DlYEGJN0w/oeuFHrhR7D+i8an752yLP6P3ztkWhys/pP3ztkWhyz+mBBiTdLxqP6cKPXCj1wo/qBBiTdLxqj+pFocrAgxKP6ocrAgxJuk/qyLQ5WBBiT+sKPXCj1wpP60vGp++dsk/rjU/fO2RaD+vO2RaHKwIP7AgxJul41Q/sKPXCj1wpD+xJul41P30P7Gp++dsi0Q/si0OVgQYkz+ysCDEm6XjP7MzMzMzMzM/s7ZFocrAgz+0OVgQYk3TP7S8an752yM/tT987ZFocz+1wo9cKPXDP7ZFocrAgxI/tsi0OVgQYj+3S8an752yP7fO2RaHKwI/uFHrhR64Uj+41P3ztkWiP7lYEGJN0vI/udsi0OVgQj+6XjU/fO2RP7rhR64UeuE/u2RaHKwIMT+752yLQ5WBP7xqfvnbItE/vO2RaHKwIT+9cKPXCj1xP73ztkWhysE/vnbItDlYED+++dsi0OVgP7987ZFocrA/wAAAAAAAAD/AQYk3S8aoP8CDEm6XjVA/wMSbpeNT+D/BBiTdLxqgP8FHrhR64Ug/wYk3S8an8D/BysCDEm6YP8IMSbpeNT8/wk3S8an75z/Cj1wo9cKPP8LQ5WBBiTc/wxJul41P3z/DU/fO2RaHP8OVgQYk3S8/w9cKPXCj1z/EGJN0vGp/P8RaHKwIMSc/xJul41P3zz/E3S8an753P8UeuFHrhR8/xWBBiTdLxz/FocrAgxJvP8XjU/fO2Rc/xiTdLxqfvj/GZmZmZmZmP8an752yLQ4/xul41P3ztj/HKwIMSbpeP8dsi0OVgQY/x64UeuFHrj/H752yLQ5WP8gxJul41P4/yHKwIMSbpj/ItDlYEGJOP8j1wo9cKPY/yTdLxqfvnj/JeNT987ZGP8m6XjU/fO4/yfvnbItDlj/KPXCj1wo9P8p++dsi0OU/ysCDEm6XjT/LAgxJul41P8tDlYEGJN0/y4UeuFHrhT/LxqfvnbItP8wIMSbpeNU/zEm6XjU/fT/Mi0OVgQYlP8zMzMzMzM0/zQ5WBBiTdT/NT987ZFodP82RaHKwIMU/zdLxqfvnbT/OFHrhR64UP85WBBiTdLw/zpeNT987ZD/O2RaHKwIMP88an752yLQ/z1wo9cKPXD/PnbItDlYEP8/fO2RaHKw/0BBiTdLxqj/QMSbpeNT+P9BR64UeuFI/0HKwIMSbpj/Qk3S8an76P9C0OVgQYk4/0NT987ZFoj/Q9cKPXCj2P9EWhysCDEo/0TdLxqfvnj/RWBBiTdLyP9F41P3ztkY/0ZmZmZmZmj/Rul41P3zuP9HbItDlYEI/0fvnbItDlj/SHKwIMSbp',
135s E             -                     'dtype': '<f8',
135s E             ?                               ^
135s E             +                     'dtype': '>f8',
135s E             ?                               ^
135s E                                   'shape': [
135s E                                       138,
135s E                                   ],
135s E                               },
135s E                               'units': 'Ang^-1',
135s E             -             },
135s E             -             'I': {
135s E             -                 'value': {
135s E             -                     'type': 'ndarray',
135s E             -                     'encoding': 'base64',
135s E             -                     'contents': 'b/CFyVRVVUBfmEwVjCJSQK62Yn/ZrU9AI0p7gy+MTEALtaZ5x4FJQD9XW7G/PEdA3EYDeAvcREASFD/G3LVCQHUCmggb3kBAXdxGA3ibPkB6pSxDHLs7QG6jAbwFQjlA2qz6XG3FNkAqqRPQRIg0QInS3uALEzNAKjqSy39IMUDtnjws1HovQAKaCBueHi1AAwmKH2MuK0C62or9ZVcpQCJseHqlDCdA0ETY8PRqJUBEaW/whUkkQD5cctwp/SJAIbByaJGdIUB798d71aogQJj6eVORKh9AKAr0iTwJHUDFOH8TCjEcQKeufJbnIRpA0NVW7C/7GECjHqLRHcQXQGlXIeUn1RZA0csollv6FECiC+pb5lQUQINRSZ2AhhNAAB3mywtwEkAh5SfVPt0RQM0eaAWGTBFAjWK5pdWQEEDP2m0XmusOQFOu8C4X8Q1A2A3bFmV2DUDDgZAsYMILQCbHndLB+gpAmMCtu3kqCkBw626e6hAJQAYq499nHAhAbRyxFp8CB0BZTGw+rs0GQDSFzmvs0gVAoYSZtn/lBEBn1edqK7YEQMtKk1LQrQNAho+IKZEEA0DQ7SWN0boCQCEf9GxWvQFAbATidf0CAUCCxeHMr+YAQLVsrS8SmgBA8Uv9vKlIAEB798d71cr+PwvvchHfif4/fSJPkq6Z/j+vlGWIY938P0BNLVvri/s/HNMTlnjA+j9VE0TdByD6P1IKur2kMfo/qaROQBNh+T+0AkNWt/r4Py7nUlxV9vc/P3RBfcuc+D8VxhaCHJT3P+cAwRw9fvY/fCx96IJ69j9ClZo90Ir1P9UmTu53KPU/Xf5D+u1r9T+w/s9hvrz0P2sOEMzR4/Q/A+yjU1e+8j/VCWgibPjyPyveyDzyB/M/JlMFo5K68j/lCu9yEV/xP1jFG5lH/vA/BaipZWv98D85l+Kqsm/wP9rGn6hs2O8/DFnd6jlp8D/f3coSnWXvP0IJM23/SvA/cCamC7H67j9bKJmc2hnuP0CH+fIC7PA/CFvs9lll7j/qPCr+74jsP0uQEVDhCOw/ZTTyecXT7T9VavZAK7DtPyBe1y/YDew/MQisHFrk6T/bwB2oUx7uP55eKcsQR+4/arx0kxiE6z8EAwgfSrTuP4nOMotQbOg/m8qisIsi6T8XvOgrSDPqP7mLMEW5tOU/aqD5nLtd6D+TxJJy9znoP7KbGf1oOOM/VMa/z7jw5T9Kea2E7pLpP+ljPiDQmec/lKRrJt/s5j+6n1OQnw3nP8VXO4pzVOM/dO52vTTF5D/IPzOID2zmP6ThlLn5xuI/7BaBsb6B6D/jxi3m54bqP9eIYBxcuuM/nMJKBRXV4D9i1ouhnGjgP1FLcyuEVeM/+b1Nf/aj5z9vZ195kJ7dP7Gmsijsot4/4NqJkpDI5j8VOUTcnEreP4f6XdiaLeE/Nxd/2xMk5T8qN1FLcyvaPwAAAAAAAAAA',
135s E             -                     'dtype': '<f8',
135s E             -                     'shape': [
135s E             -                         138,
135s E             -                     ],
135s E             -                 },
135s E             -                 'units': 'cm^-1',
135s E                           },
135s E                       },

There are similar differences for the other test cases.

The full log may be available on ci.debian.net (although techbros externalising their costs on the world with scrapers for their slop machines have meant that it might not be available so easily):

https://ci.debian.net/data/autopkgtest/testing/s390x/s/sasdata/74100932/log.gz
https://ci.debian.net/data/autopkgtest/testing/s390x/s/sasdata/

[pkienzle]

The target values are stored in test/sasdataloader/json as IEEE784 little endian base 64 encoding.

AI says S390 numpy now uses IEEE784 so it is an endian only problem.

You could have the tests pass by encoding stored values as little endian regardless of type. Update here:

sasdata/sasdata/metadata.py

Lines 600 to 607 in d7d9411

	case ndarray():
	return {
	"type": "ndarray",
	"encoding": "base64",
	"contents": base64.b64encode(obj.tobytes()).decode("utf-8"),
	"dtype": obj.dtype.str,
	"shape": obj.shape,
	}

Regardless, arrays should be converted to native endian when decoding. Update here:

sasdata/sasdata/metadata.py

Lines 513 to 520 in d7d9411

	case {
	"type": "ndarray",
	"dtype": dtype,
	"encoding": "base64",
	"contents": contents,
	"shape": shape,
	}:
	return np.frombuffer(base64.b64decode(contents), dtype=dtype).reshape(shape)

[pkienzle]

On encoding use little endian (<):

obj = np.asarray(dtype=obj.dtype.newbyteorder('<'))
return { ... }

On decoding use native byte order (=):

obj = np.frombuffer(...).reshape(shape)
obj = np.asarray(dtype=obj.dtype.newbyteorder('='))
return obj

The np.asarray() call does nothing if the dtype is already correct. The alternative obj.astype(...) always makes a copy.

I tested this for numbers of various sizes (1, 2, 4, 8) and kinds (i, u, f, c), for byte strings and for unicode strings (which are stored internally as 4 byte integers). I didn't play with M=Datetime, m=Timedelta or V=Void kinds.

I don't think github CI has any big endian architectures, so someone with access to a big endian machine should make the patch and test it.

[pkienzle]

Here's a test for boolean using sympy.isprime:

In [116]: np.asarray(data:=np.array([isprime(x) for x in range(128)]), dtype=data.dtype.newbyteorder('>')).tobytes()
Out[116]: b'\x00\x00\x01\x01\x00\x01\x00\x01\x00\x00\x00\x01\x00\x01\x00\x00\x00\x01\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\x00\x01\x00\x01\x00\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x01\x00\x01\x00\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x01\x00\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x01\x00\x00\x00\x01\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01'

In [117]: np.asarray(data:=np.array([isprime(x) for x in range(128)]), dtype=data.dtype.newbyteorder('<')).tobytes()
Out[117]: b'\x00\x00\x01\x01\x00\x01\x00\x01\x00\x00\x00\x01\x00\x01\x00\x00\x00\x01\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\x00\x01\x00\x01\x00\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x01\x00\x01\x00\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x01\x00\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x01\x00\x00\x00\x01\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01'

[llimeht]

Many thanks for that analysis @pkienzle! Much appreciated!

The following implements your suggestion with one tweak - based on the numpy docs it calls both byteswap and newbyteorder).

With this in place, tests pass on both amd64 and s390x (yes, we have developer-accessible machine of various exotic architectures to help with porting work like this; I've also seen some packages use qemu from github CI to test against different architectures).

Any comments before I turn it into a PR?

diff --git a/sasdata/metadata.py b/sasdata/metadata.py
index 588a4de..0cd2da5 100644
--- a/sasdata/metadata.py
+++ b/sasdata/metadata.py
@@ -12,6 +12,7 @@ Any useful metadata which cannot be included in these classes represent a bug in
 import base64
 import json
 import re
+import sys
 from dataclasses import dataclass, field, fields, is_dataclass
 from typing import Any
 
@@ -517,7 +518,10 @@ class MetaNode:
                     "contents": contents,
                     "shape": shape,
                 }:
-                    return np.frombuffer(base64.b64decode(contents), dtype=dtype).reshape(shape)
+                    # ensure that the data is in native byte order once read
+                    data = np.frombuffer(base64.b64decode(contents), dtype=dtype).reshape(shape)
+                    data = np.asarray(dtype=data.dtype.newbyteorder('='))
+                    return data
                 case {"value": value, "units": units}:
                     return from_json_quantity({"value": from_content(value), "units": from_content(units)})
                 case _:
@@ -598,6 +602,11 @@ class MetadataEncoder(json.JSONEncoder):
             case Quantity():
                 return {"value": obj.value, "units": obj.units.ascii_symbol}
             case ndarray():
+                if sys.byteorder == 'big':
+                    # on big-endian systems, convert the object to little-endian
+                    # to standardise serialisation, as expected by the test suite
+                    obj = obj.byteswap()
+                    obj = obj.view(obj.dtype.newbyteorder())
                 return {
                     "type": "ndarray",
                     "encoding": "base64",

[rozyczko]

Sorry to butt in, but in sys.byteorder == 'big' you assume that every array is in native order? An edge case, but still worth considering.

h5py returns arrays in whatever the file byte order. on s390x the little-endian test HDF5 files load as <f8, and the patch would byteswap them to >f8, which is probably what you didn't want?

Can't we just make both sides unconditional casts? This would handle native and non-native arrays the same way

(Just realized the code was the same snippet as @pkienzle proposed in this thread, so removed it)

[llimeht]

@rozyczko indeed - thanks!

I think you're right about that... we indeed do not want to unconditionally coerce as I've done because it might already be LE.

What I'm puzzled about is why this method is not what numpy documentation (linked above) advises for swapping.

I guess it's best to avoid the conditional and just do this, rather than extend the conditional with a obj.dtype.byteorder == ">"; it's close-enough to a no-op if the endianness is already correct.

(I also notice that no test is hitting that part of the from_json method above, and there was a mistake in the previous patch, now corrected below)

diff --git a/sasdata/metadata.py b/sasdata/metadata.py
index 588a4de..4a8cafc 100644
--- a/sasdata/metadata.py
+++ b/sasdata/metadata.py
@@ -517,7 +517,10 @@ class MetaNode:
                     "contents": contents,
                     "shape": shape,
                 }:
-                    return np.frombuffer(base64.b64decode(contents), dtype=dtype).reshape(shape)
+                    # ensure that the data is in native byte order once read
+                    data = np.frombuffer(base64.b64decode(contents), dtype=dtype).reshape(shape)
+                    data = np.asarray(data, dtype=data.dtype.newbyteorder('='))
+                    return data
                 case {"value": value, "units": units}:
                     return from_json_quantity({"value": from_content(value), "units": from_content(units)})
                 case _:
@@ -598,6 +601,10 @@ class MetadataEncoder(json.JSONEncoder):
             case Quantity():
                 return {"value": obj.value, "units": obj.units.ascii_symbol}
             case ndarray():
+                # Ensure that the data is little-endian, no matter the
+                # original format and data source; this standardises serialisation,
+                # as expected by the test suite
+                obj = np.asarray(obj, dtype=obj.dtype.newbyteorder('<'))
                 return {
                     "type": "ndarray",
                     "encoding": "base64",