Repository navigation
Conversation
Collaborator
Author
|
Figures from both examples: Central Amazon: why the climatology exists
One 8-day composite loses half the scene to cloud; the 2003–2019 composite of the same period 
The regional cycle peaks in the dry season (Aug 29) and bottoms in the cloudy wet season. The Missouri Ozarks: the class field and what it buys
The IGBP map resolves the Ozark highlands as deciduous broadleaf over the hills in a 
The filled 2015–2019 climatology: leaf-off near 0.55 in January, green-up over days 100–150, 
The 46 climatological stamps sit at their compositing-window centers, so interpolation passes 
One calendar year against its own climatology as the anchor: May 1 2019 goes from 76.1% |
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
xkykai
changed the title
xkykai
changed the title
Reading a windowed dataset file onto a fine Float32 `LatitudeLongitudeGrid` placed the field one native cell off on both axes, with no error raised. At 1/336° that is a displacement of about 300 m. Two independent causes, each on its own sufficient: - `convert_to_λ₀_λ₀_plus360` takes its modulo in the precision of its argument, so a Float32 grid node picked up 1.53e-5° of error, the spacing of Float32 near 360. The conversion is now done in double precision. - The tolerance in `compute_bounding_indices` was `eps(Float32) * |h|`, which is smaller than the grid's own node error: nodes are built by accumulating a spacing, so a latitude node can sit several ULPs from the file coordinate it should match. The tolerance is now a quarter of the file's local spacing, floored by the previous value. Adds `test/test_region_offset.jl`, which reproduces the offset with no download and fails on the previous code. Coarse-grid offsets, including ERA5's 0.25° lattice, are unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds `CopernicusVegetation`, an adapter for the C3S `satellite-lai-fapar` v4.1 record: leaf area index retrieved from Sentinel-3 OLCI and SLSTR broadband white-sky albedo with the Two-stream Inversion Package, 10-daily on a 1/336° grid covering 80°N–60°S from July 2018 to December 2024. Downloads reuse the existing CDSAPI extension, so there is no new dependency, weakdep or extension. The product reports an effective leaf area index. The global grid holds 5.7 billion cells per dekad, so a region is subset server-side through the request's `area`, filenames are keyed by date and region, and requests are batched one per calendar month. A read hyperslabs exactly the native-grid window, which pins the region offset to zero and lifts only the cells that are used; a stored file that does not span the requested region raises rather than clamping and replicating edge values. Two conventions of the product are handled explicitly: the coordinate variables label each cell's west and south edge rather than its center, and latitude is stored north to south. The `retrieval_flag` bitfield also carries a `scale_factor`, so its bits are read unscaled through `.var`; a CF-decoded read would return a lossy Float32 for a 30-bit flag. Quality screening is opt-in, and the bits cannot be chosen by their names. `unusable_retrieval_flags()` selects the three genuine rejection bits, which remove 0.02% of a mid-latitude summer scene. Of the remaining flags, `obs_inconsistent` and `obs_nosnow_only` are set on 95% and 100% of that same scene, so screening on either discards nearly the whole field. `dekadal_dates` and `native_cell_range` move into `DataWrangling` and are now shared with `CopernicusAlbedo` rather than duplicated, and the ZIP-extraction and date-identification helpers in the CDSAPI extension are generalized so both Copernicus adapters use them. Includes a module README, doctests, `Λ` in the notation appendix, a Literate example over the central Amazon, credential-free tests covering the read path, the flag helpers, filenames, the request area and the date cadence, and a network-gated download test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
xkykai
force-pushed
the
xk/lai-canopy-roughness
branch
from
July 25, 2026 00:37
0d72e64 to
fb1deaa
Compare
Over the central Amazon the regional-mean effective leaf area index reads about 1.7, where closed rainforest should give 3 to 4 even after accounting for foliage clumping. The product's quality assessment documents this as noise over equatorial forest producing unexpectedly low retrievals, on top of a substantial negative bias against ground measurements. Screening does not recover it: the low values are spread across the scene rather than concentrated in the pixels the retrieval flags mark, and applying the high-uncertainty flag lowers the regional mean from 1.70 to 1.63 rather than raising it. The example now says so, since the figures alone would understate the canopy to anyone reading them quantitatively. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ingests MCD12Q1 V061 annual land cover alongside the MCD15A2H leaf-area record, on the same 500 m lattice, and uses the class field to fill what multi-year compositing leaves behind. Also lands the MCD15A2H ingestion and its seasonal climatology, which were sitting uncommitted in the tree. - `MCD12Q1(; legend)` — `:landcover_class` under IGBP / LAI / PFT legends, plus `:quality_flag` and `:land_water_mask`. Class codes are read on the product's own grid; `class_fractions` aggregates them onto a coarser lattice as continuous per-class area fractions. - `fill_seasonal_gaps!(series, land_cover; ...)` — interpolation along time with a per-cell `class_maximum_gap`, then a same-class donor's seasonal shape scaled to each cell's own level, then the donor curve itself. Returns the filled series with a provenance code and donor reach per cell. Setting `anchor` to a climatology switches the donor to the cell's own climatological curve, which is how one calendar year is filled. - `time_average(series, bounds, window)` — overlap-weighted reduction onto a coarser cadence; 8-day composites do not nest inside months. - `gap_fill_denial` scores any of the above by withholding observed values. - `fill_gaps!` accepts a per-column `max_gap`, and errors rather than silently filling a window when a series is only partly in memory. - A climatology now skips a date the archive has no granule for (the record has outage holes, e.g. 2016-02-18) instead of aborting the build. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A product stamped at the start of an averaging window represents the middle of it, so mapping the stamp straight onto the time axis gives every temporal interpolation of that product a systematic phase lead of half a period. `native_times` now advances each date by `time_window_offset(metadatum)`, which is zero by default and so leaves every instantaneous dataset alone. MODIS 8-day composites return half their own window, which is short for the period that ends the year. The visible consequence is that the 46 stamps of a seasonal climatology span exactly one year instead of 46 x 8 = 368 days — the period a `Cyclical()` series infers and wraps on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gap-fill chain deliberately leaves the non-vegetated classes missing, because the product does not retrieve there and borrowing a neighbour's canopy would be an invention. Measured over both demo regions that residual is all of what is left — no vegetated cell runs out of donors — and those cells are not missing a value. Leaf area per unit ground area over water or tarmac is zero, and that is the value. `zero_non_vegetated!(series, land_cover)` writes it, and belongs after the fill and before the regrid: a NaN at a shoreline dilates into its neighbours through the interpolation stencil, while a zero blends into the cell mean correctly, since leaf area is already per unit ground area. Score a fill before zeroing, never after — `gap_fill_denial` samples cells that carry a value, so zeroed water would enter the sample as truth the chain reproduces exactly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The C3S `satellite-lai-fapar` record was an earlier choice of leaf-area source, superseded by MCD15A2H: MODIS is what the aerodynamic-roughness parameters were calibrated against, it reports true rather than effective leaf area so no clumping conversion is needed, and over the demo boxes the C3S record carries an unrecoverable tropical low bias. Keeping a second, unused source in the tree costs maintenance and confuses which field is the product. Removes `CopernicusVegetation`, its CDS download path, its tests and its example. The helpers that commit hoisted into shared code — `dekadal_dates` and `native_cell_range` — stay, since the albedo adapter and MODISLand both use them now. Also drops the leaf-area examples from the documentation build; they need Earthdata credentials and hours of granule downloads, so they are run by hand rather than by Documenter. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
xkykai
changed the title
xkykai
marked this pull request as ready for review
July 28, 2026 23:39
xkykai
marked this pull request as draft
July 28, 2026 23:40
Three places still pointed at the C3S record as something this repository ingests, or at an example that no longer exists. The true-versus-effective distinction and the closed-rainforest magnitude check are both worth keeping, so they stay, stated without the dangling reference. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`dekadal_dates` and `composite_dates` each encode one product's compositing cadence — day 10/20/end-of-month for Copernicus Global Land, a day-of-year-1 restart every January for MODIS. Neither is shared, so they belong with the adapter that dispatches `all_dates` on them rather than in the shared `DataWrangling` utilities, which now hold only `compute_native_date_range` and `comparable_datetime`. `native_cell_range` stays shared; the albedo adapter and MODISLand both use it. - `DataWrangling.dekadal_dates` -> `CopernicusLandAlbedo.copernicus_albedo_dekadal_dates` - `DataWrangling.composite_dates` -> `MODISLand.modis_composite_dates` Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`inpainting.jl` covers spatial nearest-neighbor inpainting and vertical continuation of fields. `fill_gaps!` and the class-aware seasonal chain built on it work along the time axis instead, and had grown to four fifths of the file. They move to `gap_filling.jl`. Two docstrings there pointed at `class_maximum_gap`, which lives in `MODISLand`, so shared code named a product-specific helper. Reworded, leaving the reference running one way: an adapter may cite the shared code, not the reverse. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…nd improve metadata integration in time averaging
- Renamed `maximum_gap` to `max_gap` in various functions and examples for consistency. - Updated related documentation and README files to reflect parameter name changes. - Enhanced error handling in `warp_modis_layer` function to ensure proper resource management. - Improved clarity in the `restrict` and `native_cell_range` functions with better documentation. - Added tests for MODIS granule downloading and compositing to ensure functionality. - Adjusted tests to align with new parameter names and improve coverage.
…data processing - Updated variable names from `Λ` to `𝒜` in various test files to improve readability and maintain consistency. - Modified the `time_average.jl` file to simplify time calculations by removing unnecessary constants and using `Dates.Second` directly. - Adjusted tests in `test_modis_land.jl`, `test_modis_land_downloading.jl`, `test_seasonal_gap_fill.jl` to reflect the new variable naming. - Ensured that all tests still pass after renaming and refactoring.
Move the MODIS granule reader into the ArchGDAL extension directory as `modis_land.jl`, following main's split of that extension into per-dataset files. Take main's window-center helpers over the branch's equivalent `time_window_offset`, keeping `sample_bounds` for `time_average`. Keep `gap_filling.jl`'s `fill_gaps!`, which supersedes the copy main left in `inpainting.jl`. Drop MODISLand's `earthdata_download` and `cmr_granules_url` for the shared ones in `DataWrangling.earthdata`. The shared `cmr_granules_url` gains a `date` keyword for products whose granules are dated, and `download_with_retries` no longer retries an interrupt. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…e-grid gap-fill safeguards
…lEarth/NumericalEarth.jl into xk/lai-canopy-roughness
Keeps both sides' ArchGDAL extension imports (MODISLand alongside OpenLandMap, write_atomically alongside the tiling helpers) and the shared majority_class! for ESA WorldCover. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
time_average becomes a DateTime method on Oceananigans.OutputReaders.time_average: it maps each calendar window onto unit length, calls the upstream reduction, and returns a FieldTimeSeries timed at the window centers in seconds from the first bound. It returns the series alone, like the upstream method, so callers build the window edges from window_bounds. Oceananigans 0.111 lacks time_average and is dropped from compat. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Hold gaps at the ends of an open series to max_gap, so an anchored single-year fill takes long edge gaps from the climatology instead of persisting the first or last observed value - Scale the donor curve over observed periods only, as the minimum_anchor_periods docstring states - Land every categorical MODIS variable (MCD12Q1 quality flag and land-water mask, MCD15A2H land-cover code) on a model grid as the majority class instead of interpolating codes, and copy them on the native grid rather than erroring - Key the fill cache on a hash of every array entry, not the sum - Name cached MODIS files by the lattice cells the region selects, so regions that round alike but select different cells no longer share a file - Examples: score the gap fill on the composited series rather than the filled one, regrid leaf area conservatively onto the coarse model grid, and correct the reducer = maximum, uncertainty and roughness claims Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Align longitude conventions, fill class-fraction halos, and launch majority-class kernels over field indices. Correct the interpretation of withheld-data scores and add area-conservation and 3D model-grid regressions. Assumes the upstream Oceananigans coordinate-aware and reduced-dimension regridding fixes. CPU MODIS tests and import checks passed with those fixes in the local Oceananigans checkout.
- Make a native-grid read a copy in one place, `regrid_from_metadata!`, so dataset `interpolate_physical!` overrides need no `same_grid` guard - Merge the open and cyclic gap scans into one loop and share one host round trip (`on_host!`) across the FieldTimeSeries methods - Build the seasonal-fill cache key only when a cache is requested - Stage downloads through `write_atomically` only, not twice - Write every MODIS lattice file through one `write_lattice_file`, used by the extension, the climatology build and the test fixtures - Drop forwarding accessors, an unused alias, `periods_per_year`, and the array-only `class_fraction`/`class_fractions`; derive legend ranges - Reuse one fraction buffer in the WorldCover majority-class regrid - Run the example's gap fill once and read stage gaps off the provenance Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Oceananigans' `regrid!` only regrids between grids of the same topology, so a class map on a native grid with a `Flat` vertical axis could not land on a model grid with a vertical axis, and `Field(::MCD12Q1 metadatum, grid)` threw for every three-dimensional grid. Regrid onto a horizontal copy of the target grid and copy the result in. The longitude relabel moves into its own helper so the next caller can share it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`fill_seasonal_gaps!` defaulted to `cyclic = true` whenever no `anchor` was given, so a dated series that is not one whole seasonal cycle had its end gaps interpolated against its other end. Default to `false`, as `fill_gaps!` does; a climatology passes `cyclic = true`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The climatology's per-period filename carried the variable, years, reducer and period but not the source dataset's `screened_flags`. A climatology built under a different screen therefore found the files of the first screen on disk and silently reused them. Put the screen in the filename. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y faces Oceananigans 0.113's `regrid!` picks a single-dimension path when two grids share a size in one dimension, regridding only the other and copying the first index by index, and throws for grids of equal horizontal size. The test pins the correct fractions, so it fails on every architecture. The fix is CliMA/Oceananigans.jl#6253, merged but unreleased and outside the current compat. Mark the checks broken so an Oceananigans release that fixes it shows up as an unexpected pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Leaf area becomes per unit land area throughout, so a raw read, a filled series and a model-grid field all mean the same thing: - `valid_area_mean!` regrids conservatively and divides by the area of the finite source cells, so missing pixels drop out of a model cell's mean rather than counting as zero or blanking the cell. MCD15A2H and `MODISLAIClimatology` variables land through it in place of bilinear interpolation, which point-sampled the 500 m lattice on coarser grids and spread every gap into its neighbors. `:landcover_code` keeps its majority class. - The product's urban, snow/ice and barren codes read as zero leaf area and FPAR, whatever the quality screen says; water, wetland and unretrieved pixels stay `NaN`. - `zero_non_vegetated!` lets the MCD12Q1 class decide: urban, snow and barren become zero and water becomes `NaN`, whatever the series held, so the land it averages over is the land `class_fractions` reports. Both MODIS examples follow the convention. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve the Project.toml compat conflict by taking main's OceanBioME 0.20 and Oceananigans 0.114, and keeping 0.111 dropped, since the branch averages calendar windows with Oceananigans' time_average (0.112 onwards). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- write_atomically creates the target directory, so a bare or nested cache path works - gap_fill_denial never writes the cache, so scoring leaves the real series' cache intact - Non-fillable columns enter the temporal fill with max_gap = 0, which fill_gaps! skips without reporting, so unfilled classes no longer inflate the unfilled-gap warning - An over-dimensioned max_gap raises the intended ArgumentError - warp_modis_layer opens granule subdatasets inside the try, so a failed open frees the rest - The calendar time_average warns in calendar units, gives the truncated last window unit length, and wraps Oceananigans' result instead of copying it; TODO to upstream it - A FieldTimeSeries anchor hashes in the same 3-D form as an array anchor - Class-fraction conservation tests use targets that share no size with the source, so they run as real tests; only the same-grid case stays broken - Register both MODIS examples in the docs build, with Earthdata credentials in CI - Derive the MODIS LAI land-cover codes from the class table, shorten the seam error, and rename cv_rmse and icols/jrows Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ions - The inpainted cache name carries the variable and, for LAI granules, the quality screen: one raw file holds every layer, so LAI, FPAR, differently screened LAI, and a climatology's retained-retrieval count shared one cache and read each other's data - MODIS declares its location through dataset_location, so a multi-date FieldTimeSeries on a 3-D grid is 2-D like the single-date Field - valid_area_mean! writes only the window of a windowed target - Drop the unused `import Oceananigans` from MODISLand Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The native-grid copy lived in both the dataset backend and the regrid entry point. Keep only the regrid path, and have tiled regridding step aside when the target is the native grid so the copy is reached. WorldCover's per-block class_fractions now extends the DataWrangling function instead of shadowing it, so loading both no longer drops one. Drop the seasonal-climatology trait on WOAMonthly, which is outside this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TYPEDSIGNATURES renders nothing on a struct docstring, so the MCD15A2H, MCD12Q1 and MODISLAIClimatology docstrings lost their constructor line. Write it by hand with its defaults, as the other datasets do. A docstring on one method shows only that method's signature, so give the array and series methods of fill_gaps!, fill_seasonal_gaps! and zero_non_vegetated!, and both methods of period_index, their own docstrings. Name the array argument `series`, as the prose does. A FieldTimeSeries is an AbstractArray, so TYPEDSIGNATURES gives the array methods the FieldTimeSeries method's return type. Render those with SIGNATURES and say in the prose that the argument is an array. Drop the duplicate DocStringExtensions import in DataWrangling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kaiyuan-cheng
force-pushed
the
xk/lai-canopy-roughness
branch
from
October 22, 2026 15:45
8317f45 to
e7fe06b
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ingests MODIS MCD15A2H leaf area index and MCD12Q1 land cover, builds a multi-year seasonal climatology from the first, and fills what compositing leaves behind with a class-aware gap fill: time interpolation, then a same-class donor curve scaled to the cell's own level, then the local class mean. Closes #650.
Shared read-path changes: a read onto the native grid copies instead of interpolating (the bilinear stencil dilates every
NaN), a one-cell region offset on fine Float32 grids is fixed on both axes,averaging_windowis defined for the MODIS composites and the ten-day CGLS albedo files, and a whole seasonal climatology no longer warns thatCyclical()wraps.Public API:
MCD15A2H,MCD12Q1,MODISLAIClimatology,build_lai_climatology!. Both datasets needArchGDALand a NASA Earthdata login.DataWrangling:fill_gaps!(gainscyclicand a per-columnmax_gap),fill_seasonal_gaps!,gap_fill_denial,window_bounds,class_fractions.time_average(fts, bounds::AbstractVector{DateTime}, window::Period): a method on Oceananigans'time_averagethat averages onto calendar windows such asMonth(1). Oceananigans compat is now 0.112, 0.113.DataWrangling.MODISLand: the class tables and helpers (igbp_class_names,class_maximum_gap,zero_non_vegetated!,period_index,retained_retrieval_metadatum,lai_screening_mask).