While we have unit tests for the implemented file loaders, they only test with a handful of data files. But there are much more data files out there, and a lot of them may not work with the current implementation.
We should be able to create some logic in the unit tests which will take in a list of files to load, and provide a subset of lines to make sure those were read in correctly. Importantly, it should be possible to scale this up to as many test files as possible, which will then make it easy to see the impact of any changes we make to the loaders (on one hand, to see how many files now load that didn't previously, and also to make sure there are no regressions in the functionality).
While we have unit tests for the implemented file loaders, they only test with a handful of data files. But there are much more data files out there, and a lot of them may not work with the current implementation.
We should be able to create some logic in the unit tests which will take in a list of files to load, and provide a subset of lines to make sure those were read in correctly. Importantly, it should be possible to scale this up to as many test files as possible, which will then make it easy to see the impact of any changes we make to the loaders (on one hand, to see how many files now load that didn't previously, and also to make sure there are no regressions in the functionality).